Ryan Oglesby

The Life and Death of a Pipeline Master

Mon, 12 Nov 2018 00:17:42 GMT

Originally posted in late 2015 or early 2016 as an internal blog post. It tells the story of introducing and later abandoning a strategy for improving build pipeline culture. The setting is a 100+ person program, distributed between India and USA, maybe 10 teams, dozens of services and apps, doing trunk-based development on a Java monorepo.

Beginnings

The time was July, 2015. It was a different time then. A wild west of sorts. We had a build pipeline, but it was horribly untrustworthy. Our user journey tests failed often, both because of broken functionality and because of our own deficiencies with asynchronous testing. The underlying infrastructure was extremely unstable. Dropped database connections happened multiple times a day.

These problems contributed to a growing cultural disease of inattention to the information coming out of our build pipeline. Developers often committed on a red build, whether it was because they did not trust it, because they felt like they could not fix it anyway, or because they assumed that someone else must be working on it. Gradually, the team was developing bad habits.

So, we felt that establishing a rotating role would give us more organization and discipline around our build pipeline hygiene. The role would be the go-to person for build hygiene issues. The Pipeline Masters were responsible for ensuring that a broken build was fixed in a timely manner, though they were not necessarily responsible for doing the actual work to fix it. Their core value was as an organizer, communicator, and enforcer. They would help identify trends of common problems so that the teams could prioritize the most important. And their power even extended as far as being able to revert any check ins when they were committed on top of an already red build.

Deterioration

What started out with good intentions began to devolve within a few rotations. One of the first things we noticed was inconsistent hand-offs from one Pipeline Master to the next, resulting in a loss of context and continuity. The tracking of persistent problem areas and common fixes died off soon.

Instead of a more ideal scenario of shared responsibility for the build pipeline, a culture began to take shape around reliance on the Pipeline Masters to enforce hygiene. Because the Pipeline Masters would send out an email titled “Build Broken - Don’t Commit,” we got into the practice of relying on that to tell us if the build is healthy instead of using radiators or other notification tooling. Not to mention, that produced a lot of extra noise in our already busy email inboxes!

All Things Come to an End

Over time, people began to ask why we needed the Pipeline Masters more and more. Shouldn’t proper build etiquette be a contract shared amongst everyone on the project? Eventually, other duties even started to creep into the Pipeline Master role. “They are already in charge of the build, so they can just do this other kind-of-pipeline-related thing too.”

What started as a way to reset build pipeline behavior had reached a point where it was no longer realizing the advertised benefits, and even becoming detrimental to the account culture. The role was also being replaced by better alternatives. We turned on build radiators. People started being more disciplined about using GoCD comments when the build was broken. The underlying infrastructure became a bit more stable. Our tests got more reliable. We even built a script that checked that status of all gating builds, which anyone could run locally before pushing.

Though our build pipeline etiquette is not perfect, we are exhibiting much more collective ownership. We have published a wiki entry with the expectations of all developers that outlines the expected contract: following the “Check In Dance,” placing comments in GoCD for communication, assuming ownership especially after checking in code, and ensuring the build is not red at the end of the day.

Final Thoughts

In retrospect, it is hard to say whether we would have ended up at the same place now if we had never implemented the Pipeline Masters. From time to time, suggestions get raised of other ways of “forcing” proper build etiquette, such as a pre-commit script that will not let you push if builds are green. I believe that these types of ideas are anti-patterns, and instead we should be focusing on the culture of the teams. It’s a people issue, not a technology issue.

Green builds for all!

Software Development is a Team Sport

Sat, 21 Apr 2018 00:00:00 GMT

What does it mean to be a member of a “software development team?” Is the team a single collective, or a collection of individuals?

When team members act like individuals who happen to be working on the same codebase at the same time, it leads to tensions that lower the team’s overall productivity and lessen the average individual’s fulfillment. A team of software developers is most effective when they value the team over the individual.

Everyone has preferences for the way they like to work and write code. That’s human. But, putting those preferences over the needs of the team is selfish and unprofessional.

Code style

Having your own code style on a software team doesn’t make sense. Code style isn’t about your preference. You don’t format code so you can write it, and you don’t format code so you can read it. You format code so others can read it. You’ll write it once, and then it will be read a thousand times by people who are not even on the team yet.

Tools that apply formatting to code automatically, such as prettier or gofmt, fit perfectly into the team sport mindset. Some people complain, saying “I don’t like the style that prettier applies.” The point is that the style doesn’t matter. What matters is that the code will be of the same style, and that you don’t have to think about it anymore. A codebase that looks like it was written by a single entity is easier to read and understand. And when you spend less time thinking and talking about code style, you can spend more time focusing on architecture, performance, and the domain.

A team values a unified code style over individual preference.

Trunk-Based development

Long-lived feature branches is a sign of a collection of individuals who call themselves a “team.” The tendency to want to work on your own isolated branch for a long period of time signifies a mindset of “this is my code over here, and that is your code over there.” Furthermore, when developers of this mindset submit their code to merge into the mainline, they tend to think “I’m done with that and now its someone else’s responsibility, time for me to work on something else”. It becomes easy to lose sight of what others are doing and what else is going on in the codebase. It can also lead to specialized knowledge silos, which further separates the team members and is a risk to the success of the project.

Trunk-Based Development supports the idea that there is one codebase that every member of the team owns equally. There is no mine and yours. There is no hand-off and change of responsibility. It is much easier to see what else is happening in the codebase simultaneously. Large scale refactoring is safer and easier. Merge hell is less likely, as long as integration with trunk is frequent. Having only one long-lived code branch provides an anchor for the team to rally around.

A team values collective code ownership over individual pride.

Pair programming

Pairing has a lot of benefits: code review happens immediately, new team members can be brought on board quickly, knowledge silos are less likely, and so on. Pairing is also taxing. Not everyone has the mental and emotional energy to pair program day in and day out. Experienced developers often don’t want to pair because they feel that they are more productive when working alone.

The interesting part is that most of the reasons is support of pair programming relate to team optimizations, while most of the reasons against it speak to the individual. While pair programming is a positive thing for most teams, it is possible to over do it, to the detriment of the happiness of the humans. So, rather than treating pair programming as an all or nothing dogma, the critical concept is that the team is in a constant state of collaboration in one way or another.

Teams that don’t prioritize the overall shared knowledge of the team perpetuate the rockstar coder stereotype that is too prevalent in the industry. By not working in a culture of collaboration, individuals are put on pedestals, seen as invaluable, and rewarded. This only hurts the team. What happens to all their specialized knowledge when they are sick, or worse, they leave the team? How long does it take new team members to ramp up when the most experienced dev is always working alone?

A team values shared knowledge over individual productivity.

Values over practices

The most productive and happy teams I’ve worked on have operated as a team, not as individuals. Those teams built a set of values shared among the team members, and derived practices in support of them.

I like being on teams that value collaboration, feedback, openness, trust, purpose, and flexibility. Those values tend to lead to practices such as Agile methodologies, Trunk-Based Development, pair programming, code quality, and face-to-face communication, but those specific practices are less important to me.

What is more important is that the team operates as a team, optimizing for its effectiveness while not sacrificing the humanity of the individuals.

Exposing CSS Hidden Complexities with React

Sat, 18 Nov 2017 00:17:42 GMT

When we create web applications we use style sheets to describe presentation, and we use JavaScript to manage all the bits of state and logic. Unfortunately, we often end up with style sheets full of complexity, such as stateful pseudo-classes, style rule overrides through media queries, and deceivingly complicated animations. But what if we could leverage the expressiveness of React to take control of all of this?

Complexity is a concern better handled with JavaScript, not style sheets. If we push complexity from style sheets into JavaScript, then it becomes more obvious and predictable, thus easier to deal with.

This post is an exploration of this hypothesis applied to 3 common CSS features:

Pseudo-classes
Media queries
Animations

I’ve also created an accompanying Github repo with full, working examples of all the snippets shown in this post.

Motivation

While evaluating CSS-in-JS libraries for my current project recently, I did a deep dive on React Native for Web. The library’s philosophy and direction really resonated with me, and like many CSS-in-JS implementations it has many benefits over traditional CSS at scale. If you haven’t seen it yet check it out!

The Style FAQs in the documentation caught my eye:

“Media Queries may not be most appropriate for component-based designs. React Native provides the Dimensions API and onLayout props. If you do need Media Queries, using the matchMedia DOM API has the benefit of allowing you to swap out entire components, not just styles.”

“Pseudo-classes like :hover and :focus can be implemented with events (e.g. onFocus). Pseudo-elements are not supported; elements should be used instead.”

Because React Native for Web relies on React Native implementations to be cross-platform compatible, it eschews some CSS features in favor of JavaScript implementations. This makes a lot of sense even if you are not trying to build a cross-platform app. So, I decided to explore this concept further, without using React Native for Web.

A “simple” button

Here is the example I’ll be using, a “simple” button component and its accompanying style sheet.

/* Button.css */

button {
  display: inline-block;
  min-width: 200px;

  background-color: #2892d7;
  transition: background-color 0.5s;
}

button:hover {
  background-color: #1d70a2;
}

@media (max-width: 767px) {
  button {
    display: block;
    width: 100%;
  }
}

// Button.js

import React from "react";

import "./Button.css";

export default ({ children }) => (
  <button>{children}</button>;
)

Full component: https://github.com/ryanoglesby08/exposing-css-hidden-complexities/tree/master/src/components/CssButton

Goodbye pseudo-classes, hello setState

First up is the :hover pseudo-class. One of the key tenants of React and other UI libraries is that they pull state out of the DOM and into your components so that it is easier to reason about. CSS Pseudo-classes are a direct contradiction to this advantage. In order for the button to change it’s background color, something must know whether or not the mouse is currently “hovering” over the button. The DOM is doing that. The DOM is managing an implicit state of “mouse is over the button” or “mouse is not over the button”. The component should be managing that state, not the DOM.

So, I’ll refactor the Button component to use the onMouseEnter and onMouseLeave events and React’s setState.

/* Button.css */

button {
  display: inline-block;
  min-width: 200px;

  transition: background-color 0.5s;
}

@media (max-width: 767px) {
  button {
    display: block;
    width: 100%;
  }
}

// Button.js

import React, { Component } from 'react'

import './Button.css'
const bgColors = {
  default: '#2892d7',
  hover: '#1d70a2',
}

export default class Button extends Component {
  state = {
    hover: false,
  }

  toggleHover = () => {
    this.setState({ hover: !this.state.hover })
  }

  render() {
    const { children } = this.props

    return (
      <button
        onMouseEnter={this.toggleHover}
        onMouseLeave={this.toggleHover}
        style={{
          backgroundColor: this.state.hover ? bgColors.hover : bgColors.default,
        }}
      >
        {children}
      </button>
    )
  }
}

Full component: https://github.com/ryanoglesby08/exposing-css-hidden-complexities/tree/master/src/components/HoverButton

A more power media query

Media queries suffer from the same contradiction as pseudo-classes. There is an implicit state of “the media query is matched” or “the media query is not matched” that should be managed by JavaScript.

While pure CSS media queries are powerful, they tend to result in overriding of previously defined style values, which can often cause unwanted side effects and mistakes due to specificity errors. As the React Native for Web docs point out, they also are constrained to adjusting styles only. They can’t be used to swap out entire components.

Next, I’ll move the media queries out of the style sheet, preferring the window.matchMedia DOM API instead. react-media is a nice little package that provides a component interface for window.matchMedia.

/* Button.css */

button {
  transition: background-color 0.5s;
}

.inline {
  display: inline-block;
  min-width: 200px;
}

.fullWidth {
  display: block;
  width: 100%;
}

// Button.js

import React, { Component } from 'react'
import Media from 'react-media'

import './Button.css'

class Button extends Component {
  // same as previous Button.js snippet
}

export default ({ children }) => (
  <Media query={{ maxWidth: 767 }}>
    {matches =>
      matches ? (
        <Button className="fullWidth">{children}</Button>
      ) : (
        <Button className="inline">{children}</Button>
      )
    }
  </Media>
)

Full component: https://github.com/ryanoglesby08/exposing-css-hidden-complexities/tree/master/src/components/ResponsiveButton

Animations as components

The final thing to tackle here is the transition. For all but the most experienced CSS wizards out there, CSS animations or transitions can be really challenging. Moreover, they require the combination of multiple style rules working together.

In this case, transition: background-color 0.5s describes a state change of the property background-color. When the background-color property of the button changes to another value, either through a class being added or removed, or a style property being edited, the background will shift from its previous value to its new value over a timespan of 0.5 seconds. It’s pretty complex, and this is a very simple transition.

Finally, I’ll bring the transition into the component with the react-transition-group package, which lets you describe a transition from one component state to another over time with a simple declarative API.

// BgColorShift.js

import React from 'react'
import { Transition } from 'react-transition-group'

const defaultStyle = (startBgColor, timeout) => ({
  transition: `background-color ${timeout}ms`,
  backgroundColor: startBgColor,
})

const transitionStyles = endBgColor => {
  return {
    entering: { backgroundColor: endBgColor },
    entered: { backgroundColor: endBgColor },
  }
}

export default ({ start, end, children, ...rest }) => (
  <Transition {...rest}>
    {status => {
      const styles = {
        ...defaultStyle(start, rest.timeout),
        ...transitionStyles(end)[status],
      }

      return children(styles)
    }}
  </Transition>
)

// Button.js

import React, { Component } from 'react'

import BgColorShift from './BgColorShift'

import './Button.css'
const bgColors = {
  default: '#2892d7',
  hover: '#1d70a2',
}

class Button extends Component {
  // ... setState stuff

  render() {
    const { className, children } = this.props

    return (
      <BgColorShift
        timeout={500}
        in={this.state.hover}
        start={bgColors.default}
        end={bgColors.hover}
      >
        {bgColorStyles => (
          <button
            onMouseEnter={this.toggleHover}
            onMouseLeave={this.toggleHover}
            style={bgColorStyles}
          >
            {children}
          </button>
        )}
      </BgColorShift>
    )
  }
}

// export default with the Media query component

Full component: https://github.com/ryanoglesby08/exposing-css-hidden-complexities/tree/master/src/components/AnimatedButton

I can write tests for this stuff now!

One of the big wins as I moved complexity into JavaScript was the increased level of testability. There weren’t any tests that really made sense when all the UI behavior was implemented in a style sheet. Writing tests for these things takes some of the burden off of manual visual checks, creates documentation of the visual language, and helps prevent future regressions.

Here are some of the tests:

// Button.spec.js

it('changes background color on hover', () => {
  const button = mount(<Button>Press me</Button>)

  button.simulate('mouseEnter')
  expect(button.find('button')).toHaveStyle('backgroundColor', '#1d70a2')

  button.simulate('mouseLeave')
  expect(button.find('button')).toHaveStyle('backgroundColor', '#2892d7')
})

it('is full width on small viewports', () => {
  // Setup mock for `window.matchMedia` because test is running in jsdom
  const matchMedia = jest.fn()
  matchMedia.mockImplementation(() => ({
    matches: true,
    addListener: () => {},
    removeListener: () => {},
  }))
  global.matchMedia = matchMedia

  const button = mount(<Button>Press me</Button>)

  expect(button.find('button')).toHaveClassName('fullWidth')
})

My insights

Maybe the most obvious “downside” here is that the total amount of code has increased, by a lot. I’ve written less CSS, but much more JavaScript, and I’ve added 2 third-party libraries… just to make a button.

Let’s be clear here, my stated goal was not to create the most clever, most terse button. It was to expose complexities hidden in CSS so that I could identify complexity, reason about it more easily, control it, and test it. React actually has similar goals. React has never said that it’s goal is to produce solutions with minimal lines of code. Rather, it aims for declarative components, predictability, and JavaScript-controlled logic (https://reactjs.org/). The most clever solution is often the least understandable.

This is what good real world React code often looks like. Nothing wrong with it being verbose and explicit. Don’t mistake terse for clean! https://t.co/qjB685FnuJ
— Dan Abramov (@dan_abramov) November 17, 2017

Pros and cons

If you regularly use your browser’s dev tools to debug/play with the styles of your components, the JavaScript approach changes that a bit. For example, most dev tools let you trigger a pseudo-class such as :hover. That will no longer have a visible effect. Instead, use the React dev tools to programmatically interact with your components.

I get really nervous when I see CSS or Sass that has a lot of nesting and overriding of style rules. I like how the final CSS file I ended up with was flat. All style rules are static, there is no nesting, and nothing is overridden by another class. Ahhhhh, peace.
I’m pretty excited about encapsulating common visual concerns into reusable components with a single responsibility. In this exploration I identified a reusable media query component and an animation component that could be extracted. In another project I’ve started extracting layout components such as Box to control spacing, and Flexbox to control 1-dimensional layout. Check out these libraries for more: https://sapegin.github.io/react-spaceman/ and https://github.com/jxnblk/reflexbox.
Finally, I was not able to write a test for the transition that I was completely satisfied with. Transitions and animations are inherently time based, which does not play nicely with unit testing. The best I could do was take before and after snapshots and then rely on manual visual checks to make sure the transition looked proper while it was running. But, with a reusable BgColorShift component now at my disposal, I can now trust it’s behavior, making future transitions easier to implement.

The State of CSS

Thu, 25 May 2017 16:34:35 GMT

It’s no secret that we are currently experiencing a Cambrian explosion of innovation by the JavaScript community. But what you may not be aware of are the equally evolutionary innovations under development by the CSS community.

CSS has quietly been improving by leaps and bounds lately. So much so, that you may not even recognize it much anymore. While you weren’t looking, the CSS community has developed real solutions to many of the oddities, hacks, and deficiencies that made CSS the black sheep of the front end family in the past.

In this post I’ll reveal 5 old-fashioned CSS things that you don’t need to do anymore, and what you should do instead.

You don’t need a naming convention.
You don’t need float.
You don’t need a grid framework.
You don’t need a preprocessor.
You don’t need CSS.

You don’t need naming conventions.

And we begin with the number one problem with CSS, especially at scale: everything is global.

Source: https://speakerdeck.com/vjeux/react-css-in-js

Because all selectors exist in the same global namespace, over time it becomes very likely that you’ll run into unexpected side effects such as selectors targeting elements that you didn’t intend, or selectors being overridden by other selectors. So, we have side-stepped these issues with methodologies (OOCSS, SMACSS, or the more popular BEM) that define naming conventions to help us avoid class name collisions.

While we can’t discount the importance of these methodologies, they are merely a workaround. They don’t actually solve the problems. We needed something more. And then this happened.

Source: https://medium.com/seek-blog/the-end-of-global-css-90d2a4a06284

Mark Dalgleish and others have since pioneered the concept of locally scoped CSS, which led to the development of CSS Modules, and which has now been incorporated into modern tooling, most notably webpack’s css-loader.

// webpack.config.js

{
  test: /\.css$/,
  use: [
    {
      loader: 'css-loader',
      options: {
        modules: true,
        localIdentName: '[path][name]__[local]--[hash:base64:5]'
      }
    }
  ]
}

This is huge. When using CSS Modules, name your classes however you like. At build time, all local selectors are replaced by unique identifiers, completely removing the need for a global naming convention. Check out the docs above for CSS Modules for a deeper dive.

While this technology was born from component-oriented front-end development (think React), you can also get support for CSS Modules everywhere.

You don’t need float.

Float has managed to become both the positioning property of choice and the bane of many a developer’s existence by weaseling its way into everything from helper classes to entire grid systems. While occasionally using float to push an element left or right doesn’t get you into too much trouble, pervasive use of it for full page layouts leads to odd behavior that is a pain to figure out and fix, often requiring hacks. Pretty soon you find yourself copying the admittedly elegant, yet unintuitive “clearfix” onto every element and hoping for the best.

Thankfully, CSS3 introduced a new layout module called flexible boxes, or Flexbox for short. Flexbox is a mostly 1-dimensional layout system that ensures elements behave predictably on various screen sizes and devices. It also solves the common, yet previously frustrating task of vertical and horizontal centering. When you switch to Flexbox, you no longer have any use for float.

See the Pen GmbYjd by Ryan Oglesby (@ryanoglesby08) on CodePen.

While Flexbox is fully supported by all modern browsers, you will run into issues on older ones. So, if you have to support IE < 10 you will likely still need to fall back to floats, tables, or some other layout method.

Some browsers also need vendor prefixes, so I recommend using Autoprefixer to get the most coverage.

You don’t need a grid framework.

Everyone uses a grid. It’s a crucial part of every CSS codebase, letting you build complex full-page layouts by aligning elements along rows and columns. While rolling your own simple grid using traditional techniques is fairly straightforward, no one does it. Instead, we reach for one of the many grid frameworks, such as the original 960gs or the Sass-based Bourbon Neat.

Finally, as of March 2017, most browsers added support for the CSS Grid Layout, a 2-dimensional layout system built directly into CSS. Since most traditional grid frameworks still rely on float, CSS Grid is an attractive alternative. CSS Grid makes it easy to define your own grid without the help of a framework by providing an intuitive interface for defining the number and sizes of rows and columns, the gutter spacing, alignment of elements within the grid, named areas, and more.

See the Pen pwzNdB by Ryan Oglesby (@ryanoglesby08) on CodePen.

At the time of writing this article, CSS Grid is supported by all modern browsers, with partial support on IE if you use vendor prefixes (use Autoprefixer).

You don’t need a preprocessor.

Undeniably, CSS language extensions such as Sass and Less brought game-changing features to CSS, including data reuse with variables, better organization with nesting, and property reuse with mixins. But, these languages didn’t change anything about the core language of CSS; they added a layer of abstraction on top of it. In the case of Sass, you also need an additional binary to compile the Sass files into CSS. When using one of these preprocessors, you now have to learn CSS and the preprocessor specific syntax.

But, while most developers were blissfully coupling to their preprocessors, CSS was quietly progressing. Modern browsers now have native support for CSS variables and calc(). Other features such as color modifications, and mixins are not far behind.

:root {
  --spacing: 1.5em; /* declaring a variable */
  /* declaring a mixin */
  --base-colors: {
    color: #fff;
    background-color: color(#fff shade(+80%)); /* modifying a color */
  }
}

.some-class {
  padding: var(--spacing);
  width: calc(100% - var(--spacing)); /* dynamically calculating a value */
  @apply --base-colors;
}

Fortunately for CSS, the JavaScript community has already had the problem of browsers not supporting new features fast enough, which led to Babel, giving JavaScript developers everywhere access to “next generation JavaScript, today.” Similarly, PostCSS is a “tool for transforming CSS with JavaScript”, which gets its power from the extensive list of community maintained plugins. CSS developers can even “use tomorrow’s CSS syntax, today” with the cssnext PostCSS plugin.

Writing plain CSS and transforming it with PostCSS plugins produces more future-proof CSS with less cognitive and technical overhead than old preprocessors.

You don’t need CSS.

Ok, so that’s not entirely true. While CSS remains the only document styling language that browsers understand, you can style web applications without ever writing a single .css file. With the help of UI libraries such as React or Vue.js, modern web application architecture has embraced loosely coupled, highly cohesive components that often co-locate HTML, CSS, and JavaScript in the same file. Interestingly, this idea is in direct contradiction to previously held best practices around separation of these concerns. (This conference talk, called “Rethinking best practices” gives a great explanation.)

Source: https://speakerdeck.com/didoo/let-there-be-peace-on-css

From a CSS perspective, this rethinking has even been given a catchy name: CSS-in-JS.

At first, CSS-in-JS libraries used the HTML “style” attribute to attach styles onto elements, a flawed approach because the “style” attribute can’t do everything CSS can do. Now, most libraries generate real style sheets with the full power of CSS, eschewing use of the “style” attribute for the most part.

Writing CSS using JavaScript addresses most of the other points I’ve brought up in this article. Most new libraries scope styles automatically, make reuse of classes easy and efficient, identify and inline critical-path CSS for faster initial load times, and enable real modular sharing of open-source CSS packages. This must-read article outlines the architectural benefits for CSS-in-JS and is the best I’ve read so far.

As this space is new and exciting, there has been an explosion of libraries (this list is even slightly old now). This is a good thing. It will take some time to settle down as the community discovers best practices, such as avoidance of the “style” attribute, proper support for media queries, and the ability to use proper CSS syntax instead of a JavaScript object.

JSS is a very popular choice, with an active community and a stable interface; but I’m especially excited about the direction of styled-components, which cleverly attaches CSS to components via tagged template literals.

Source: https://www.styled-components.com

CSS is dead. Long Live CSS.

You don’t need naming conventions because locally-scoped styles by default are now baked-in to libraries.
You don’t need float because Flexbox is more powerful and less hacky.
You don’t need a grid framework because there’s one built into CSS.
You don’t need a preprocessor because CSS has more advanced features now.
You don’t need CSS because writing styles in JavaScript enables more architectural and community benefits.

I’m excited about the large-scale adoption of modern JavaScript techniques and UI libraries, especially React. But, I’m frustrated at the same time because those same codebases continue using dated, flawed CSS techniques and technologies instead of embracing modernity. It’s time we recognize the importance of these new developments in the CSS community and find a way out of our old, comfortable habits.

Conceptualizing How a Modern Single Page App is Served

Thu, 11 May 2017 13:30:08 GMT

I’ve noticed some confusion in how Single Page Applications are served among developers who are new to these apps. This post is a short explanation of how a Single Page Application gets from the web server to a user’s web browser.

First, what is a Single Page Application?

A Single Page Application (SPA) is a web application or web site that fits on a single web page, with the goal of providing a user experience similar to that of a desktop application.

All necessary code – HTML, JavaScript, and CSS – is retrieved with one page load. The page does not reload at any point, nor does control transfer to another page. As the user navigates around the application, the URL shown in the browser can be updated to provide the perception and navigability of separate logical pages in the application, but no more page loads occur unless the user forces one by clicking the browser’s refresh button or by typing a new address into the URL bar and pressing the <enter> key.

What might this look like?

This is the only HTML file for the entire application, and the only file returned to a client who requests the application. Both the app.css file and the app.js file contain all the necessary CSS and JavaScript that any part of the app needs.

<!-- index.html -->

<!DOCTYPE html>
<html>
<head>
  <title>A single page app</title>

  <link rel="stylesheet" type="text/css" href="app.css" />
</head>
<body>
  <div id="app_root"/>

  <script type="text/css" src="app.js"></script>
</body>
</html>

In this example, app.js could be a React application that produces rest of the DOM elements when it is evaluated.

What about interaction with the web server?

The sequence diagram below shows how and when the front end SPA interacts with the back end web server.

The UI server below has one and only one job: deliver the single page and all assets that the application needs when a browser requests them. Therefore, the UI server must be configured as follows:

Serve static assets. Handle any request for a static file, such as a JavaScript file, a CSS file, an image, or a font file, by loading it from the filesystem and returning it.
Match all other routes. The UI server uses a “match all” strategy so that any received request that does not match a static asset will be handled by returning the single page of the application, usually called index.html. This is necessary because the user could force a page load while on a specific URL within the application, as shown in the diagram below.

It’s important to remember that while the SPA is primarily written in JavaScript, there is no requirement that necessitates a JavaScript-based UI server. Any web server that is able to handle the configurations described above will work fine, such as Apache, NGINX, or even Amazon S3!

An Exploration of JavaScript Builders

Fri, 03 Mar 2017 09:40:44 GMT

Builders seems to be a popular topic for me, as I’ve blogged about the builder pattern before. This time, however, I look at builders through a JavaScript lens. In this post I’ll explore three techniques for modeling builders in JavaScript, specifically ES6, utilizing some of the interesting features of the modern language.

All the examples I show in this post show builders that are only for testing purposes. While the builder pattern can be used in production code, the builders shown below are not appropriate for production code because they contain pre-canned default values. Leave these in your test suite.

The Classic

I start with a simple, recognizable builder that uses “withers” to build up the object’s state. Returning this from each “wither” enables chaining. The build method will return the final plain JavaScript object.

class ProductBuilder {
  constructor() {
    this.name = 'A Product'
    this.price = 9.99
    this.category = 'other'
  }

  withName(name) {
    this.name = name
    return this
  }

  withPrice(price) {
    this.price = price
    return this
  }

  withCategory(category) {
    this.category = category
    return this
  }

  build() {
    return {
      name: this.name,
      price: this.price,
      category: this.category,
    }
  }
}

console.log(
  new ProductBuilder()
    .withName('Harry Potter')
    .withCategory('book')
    .build()
)
// =>
//    {
//      name: 'Harry Potter',
//      price: 9.99,
//      category: 'book'
//    }

Simple. Familiar. But wordy. With only three fields our builder is already quite large, and needs a lot of boilerplate. The size of the builder grows linearly with the number of fields.

While this technique is fine, it takes a very Java-esque approach, ignoring other powerful JavaScript features. Let’s investigate those.

Generate Builders with Metaprogramming

Let’s address some of the shortcomings of the previous example, namely the repetition of the “withers.” Rather than manually typing out each “wither” method, let’s generate them automatically.

class ProductBuilder {
  constructor() {
    this.name = 'A metaprogrammed product'
    this.price = 9.99
    this.category = 'other'

    // Generate "wither" methods for each property
    Object.keys(this).forEach(key => {
      const witherName = `with${key.substring(0, 1).toUpperCase()}${key.substring(1)}`
      this[witherName] = value => {
        this[key] = value
        return this
      }
    })
  }

  build() {
    // Get an array of all non-function properties of this builder
    const keysNoWithers = Object.keys(this).filter(key => typeof this[key] !== 'function')

    // Transform the array of keys into an object
    return keysNoWithers.reduce((returnValue, key) => {
      return {
        ...returnValue,
        [key]: this[key],
      }
    }, {})
  }
}

console.log(
  new ProductBuilder()
    .withName('Harry Potter')
    .withCategory('book')
    .build()
)
// =>
//    {
//      name: 'Harry Potter',
//      price: 9.99,
//      category: 'book'
//    }

This pattern produces an equivalent result as the first example. In the constructor, we automatically generate all the “wither” methods from the properties of the object. Then, in the build method, we use the properties of the object again to produce the resulting object. Some of the cool JavaScript features we use are Object.keys, reduce, and the proposed object rest spread.

While at first this may seem overly complicated (for a single builder, yes, it is), the real power is truly recognized when you have many builders. We can easily extract the generalized parts of this into a common superclass, making it extremely easy to create new builders.

class BaseBuilder {
  init() {
    Object.keys(this).forEach((key) => {
      const witherName = `with${key.substring(0,1).toUpperCase()}${key.substring(1)}`;
      this[witherName] = (value) => {
        this[key] = value;
        return this;
      };
    });
  }

  build() {
    const keysNoWithers = Object.keys(this).filter((key) => (
      typeof this[key] !== 'function'
    ));

    return keysNoWithers.reduce((returnValue, key) => {
      return {
        ...returnValue,
        [key]: this[key]
      };
    }, {});
  }
}

class ProductBuilder extends BaseBuilder {
  constructor() {
    super();

    this.name = 'A metaprogrammed product';
    this.price = 9.99;
    this.category = 'other';

    super.init();
  }
}

class SandwichBuilder extends BaseBuilder {
  constructor() {
    super();

    this.meat = 'ham';
    this.cheese = 'swiss';

    super.init();
  }
}


console.log(
  new ProductBuilder()
    .withName('Harry Potter')
    .withCategory('book')
    .build()
);
console.log(
  new SandwichBuilder()
    .withMeat('Roast Beef'
    .withCheese('Havarti')
    .build()
);
// =>
//    {
//      name: 'Harry Potter',
//      price: 9.99,
//      category: 'book'
//    }
// =>
//    {
//      name: 'Roast Beef',
//      cheese: 'Havarti'
//    }

By adopting this standard structure built on top of ES6 inheritance, we benefit from the predictability of the common interface and from the simplicity in creating new builders. Our builders are also still open for extension, as we can easily insert additional methods to add custom functionality as needed.

But, we are still following the verbose “wither” syntax. This approach is inherently object oriented. While there is nothing wrong with OOP, JavaScript is flexible and adopts multiple paradigms. What if we approach builders using a more functional programming technique?

Just Use Functions

Let’s get rid of the classes. Let’s get rid of the “withers”. Let’s model our builder as a function instead.

const buildProduct = (overrides = {}) => {
  const defaults = {
    name: 'A functional product',
    price: 9.99,
    category: 'other',
  }

  return { ...defaults, ...overrides }
}

console.log(
  buildProduct({
    name: 'Harry Potter',
    category: 'book',
  })
)
// =>
//    {
//      name: 'Harry Potter',
//      price: 9.99,
//      category: 'book'
//    }

That’s it! Here, we rely heavily on the awesome object rest spread proposal to perform merging of two objects.

While elegant, the most obvious deficiency is that there is no control over the contents of the overrides object, which could lead to mistakes. In the above example, if I were to do buildProduct({nme: 'Harry Potter'}) I would end up with a product with the default name, and an additional property “nme.” This is not ideal and could be frustrating at the least. Let’s fix that.

import { difference } from 'lodash';

const buildProduct = (overrides = {}) => {
  const defaults =
    name: 'A functional product',
    price: 9.99,
    category: 'other'
  };

  // Prevent extra keys from being introduced by the overrides
  const extraOverrides = difference(
    Object.keys(overrides),
    Object.keys(defaults)
  );
  if (extraOverrides.length > 0) {
    throw new Error(`Invalid builder! ${extraOverrides}`);
  }

  return {...defaults, ...overrides};
};

console.log(
  buildProduct({
    name: 'Harry Potter',
    category: 'book',
    keyThatDoesNotExist: 'oops'
  })
);
// => Error: Invalid builder! keyThatDoesNotExist

By using the lodash library’s array diffing utility, we can prevent this problem. And, as with other bits of reusable code, we could extract this into a function used by many builders.

In all of these examples the product object is a toy. With a larger, more complex object containing multiple levels of nested objects, this functional technique could get out of control is not properly modeled. While you could go for deep merging, I would shy away from that as it will hard to reason about. Instead, I recommend composing many builders together.

import { difference } from 'lodash'

const preventExtraOverrides = (defaults, overrides) => {
  const extraOverrides = difference(Object.keys(overrides), Object.keys(defaults))

  if (extraOverrides.length > 0) {
    throw new Error(`Invalid builder! ${extraOverrides}`)
  }
}

const buildProductName = (overrides = {}) => {
  const defaults = {
    name: 'A product',
    description: 'A product description',
  }

  preventExtraOverrides(defaults, overrides)

  return { ...defaults, ...overrides }
}

const buildProductPrice = (overrides = {}) => {
  const defaults = {
    price: 50,
    taxRate: 0.08,
  }

  preventExtraOverrides(defaults, overrides)

  return { ...defaults, ...overrides }
}

const buildProduct = (overrides = {}) => {
  const defaults = {
    name: buildProductName(),
    price: buildProductPrice(),
    category: 'other',
  }

  preventExtraOverrides(defaults, overrides)

  return { ...defaults, ...overrides }
}

console.log(
  buildProduct({
    name: buildProductName({ name: 'Harry Potter' }),
    price: buildProductPrice({ price: 9.99 }),
  })
)
// =>
//  {
//    name: {
//      name: 'Harry Potter',
//      description: 'A product description'
//    },
//    price: {
//      price: 9.99,
//      taxRate: 0.08
//    },
//    category: 'other'
//  }

By composing builders we can easily reason about our object because we have broken it down into small, independent pieces. Each builder follows a similar pattern, but is self contained, which allows for customization in specific situations when needed.

So which one is best?

Well, it depends. Yep, not what you wanted to hear right? Of course it’s never that easy. Your unique situation may make some patterns more desirable than others. Are you on a team full of ex-Java devs who are not used to JavaScript semantics yet? Maybe starting with the classic approach and moving towards the metaprogramming technique eventually will suit you. Is your team ga-ga over functional paradigms? Reach for the functional approach.

Overall, remember that JavaScript is a multi-paradigm language, capable of flexing to suit many needs. Enjoy!

It's not magic, you just don't understand it yet

Sat, 19 Nov 2016 14:09:35 GMT

Dev 1: “Wait, how did it do that?”
Dev 2: “I have not clue, it’s Spring* magic.”
Dev 1: “Oh well, as long as it just works!”
* substitute Spring for any library or framework

Every time I hear (or, admittedly, say) something like this, I cringe.

When we refer to code as magic, we perpetuate the mindset that some code is unaccessible. That we don’t need to know how something works, only that is does work. This is not the mindset of a true craftsman. A craftsman seeks to understand the tools she uses, because she knows that she will be able to use them more effectively, and will be more able to deal with inevitable failures.

Of course it is natural to think that something you have not yet understood is magic. Language features such as Metaprogramming in Ruby or Java’s Reflection API are advanced features that are often called magical. However, under examination they can be understood and are usable, important parts of those languages. If we write-off things we don’t understand, or have not tried to understand, as magic, we are doing ourselves a disservice.

A real consequence of this is the reflex to grab a library/plugin/framework for everything. Many times the simplest and most flexible thing to do is implement something yourself. At some point you may realize that you need a library to avoid having to maintain a lot of code yourself, and when that happens, switch to it. But now you will have a good understanding of what it provides. Just the other day I had a conversation with a couple developers on my team who were fixing the formatting of currency values in our JavaScript app. They had found a library online for it. Instead, I encouraged them to start with using toFixed, a native JavaScript method. Some day we may want the library, but until then the devs know a simple way for formatting a JavaScript number using fixed-point notation, and our app does not pull in yet another dependency.

Seek to understand how magic bits of code actually work. The easiest thing to do is to read the docs! I’m constantly surprised at how many developers breeze over documentation, missing key features. Take time to go through documentation, you’ll likely find some great things in there that you have not been doing.

Another way to understand is to imagine the simplest version of a library of framework. Take away all the bells and whistles and get to the core of what the library does. How do you think it works in its most basic form? Could you code a bare-bones version of it? Give it a shot! Don’t worry about “getting it right” or producing the most efficient, elegant solution. Just see if you can get something working, even a little bit. This is one of the most powerful ways to reveal the magic ways of a library or framework. You will likely be pleasantly surprised at the simplicity of your solution. And, you will likely learn a thing or two along the way.

So, in the process of writing this post, I decided to do just that. I took a stab at implementing a simple version of Spring’s autowired dependency injection facility. I spent a few hours on it, and managed to learn a few things in the process.

Detecting classes on the classpath is hard. I thought there was a simple Java API for doing this, but there does not seem to be. I ended up using some code I found online for this part so I wouldn’t have to re-implement it myself.
I got a refresher in Java Reflection as my solution needed to be able to detect the presence of specific annotations, get constructor parameters, and dynamically construct objects.
The simplest solution I could do did all the dependency injection right away, though I’m not sure if Spring actually does it this way or not. I am assuming it takes a lazy approach, which could be more efficient for program start up. Spring may even allow you to configure the dependency injection as eager or lazy.

Here is my solution: https://github.com/ryanoglesby08/spring-autowired-revealed

Try to avoid using the term “magic” when talking about code. Remember, there is no magic. Just code.

On Injecting a JavaScript Environment

Fri, 04 Nov 2016 16:44:20 GMT

Updated 2017-02-12: Added a strategy for loading configuration at initial page load, using an “env.js” file.

Recently, I needed to inject some configuration variables into my client-side app. Because Googling for it turned up a lot of “well, you could maybe do this…” I decided to write down what I see as the 3 main options:

Best in show: Load configuration at run time.
Prize for simplest solution: Build your application with configuration variables as constants at compile time.
Runner up: Use server-side templating to inject configuration at render time.

So, what do I want to do again?

First, let’s define the problem clearly. Handling configuration variables is something most applications need. For server-side applications, there are plenty of options out there. Most involve loading properties from files and then overriding values with environment variables injected at run time - such as the config package for node servers.

Client-side applications present some challenges. Because the app is delivered to the client web browser, loading files from the server does not work as easily. Specifying values with environment variables is also different, because the “environment” in a client app is the user’s web browser. We don’t have access to that JavaScript environment to inject values.

Finally, I want to follow the recommendations for configuration of the Twelve-Factor app, which boil down to:

“strict separation of config from code”
“…stores config in environment variables” rather than environment specific config files

So, let’s examine the options, and consider the pros and cons of each.

Load configuration at run time

This is strategy I went with, and seems to be working well. With this approach, the client app loads configuration from a remote server on initial page load.

Because we do not have to bake in the configuration values to the compiled JavaScript, (one of the other options) we get to build our application only once. This means we deploy the exact same code to all environments. Score.

We also do not have to use any server side templating, which gives us more freedom in how we deploy and serve our application. We can serve our application as static files if we want to.

The trick here is to build a little bit of smarts into our web server, so that it converts an injected set of environment variables to a JavaScript file on startup, and then serves it as a static file.

Here is the gist:

In your index.html file, add a script tag for an “env.js” file, which does not exist. (yet)
In your application server code, write the “env.js” file on startup using an environment variable as the contents. Then statically serve the directory containing the “env.js” file.
Make the environment variable available to the server when running it.

<!-- 1. index.html -->
<html>
<body>
  <script type="text/javascript" src="/env.js"></script>
  <script type="text/javascript" src="/app.js"></script>
</body>
</html>

// 2. server.js

const express = require('express')
const path = require('path')
const fs = require('fs')
const port = process.env.PORT || 8080
const app = express()

fs.writeFileSync(
  `${__dirname}/config/env.js`,
  `var config = ${process.env.CLIENT_ENV};`
)

app.use(express.static(__dirname + '/dist'))
app.use(express.static(__dirname + '/config'))
app.get('*', function(request, response) {
  response.sendFile(path.resolve(__dirname, 'index.html'))
})

app.listen(port)

# 3. running the server

export CLIENT_ENV='{API_URL: "http://dev.my-api.some-domain.com"}'
node server.js

One of the downsides here is that we are creating global state, the config variable, which is something that is generally frowned upon and avoided in modern JavaScript apps (and most applications for that matter). There are a few ways to address this, with the simplest being to agree not to read from the config variable directly, and create a function that your application always uses to access it. This way, the only place in your codebase where the config variable is touched is that function. You should also never modify the config variable. Treat it as read only.

An experiment - asynchronous configuration loading

Along similar lines, I also played with loading configuration from a remote server with an XMLHttpRequest. This feels like a cool approach, but the wrench here is that introducing asynchronous network calls introduces un-necessary complexity. Now, all of a sudden your config properties are not available until that call comes back. This can cause some awkward uses based on how/when you need your environment config.

To play with this idea, I wrote this unreleased npm package. Check it out if you are interested and provide feedback or make a pull request! xhr-env-provider

Build your application with configuration variables as constants at compile time.

This is probably the most straight-forward option, and uses your build tool to inject constants into your code as global variables at compile time. The DefinePlugin is the best way to do this if you are bundling your app with webpack.

## building your app

export API_URL="http://dev.my-api.some-domain.com"
webpack --define process.env.API_URL="'$API_URL'"

// inside your application code

const apiUrl = `${process.env.API_URL || 'http://localhost:8080'}`

The downside with this approach is that if your configuration varies for each deployed environment (which it surely will), you will have to build multiple versions of your app. This is less than ideal, as it adds extra steps to your build and deploy process. It also means that the asset you use in lower environments (dev) is not the same as the asset you use in production, because you have re-compiled it.

However, it is likely that you will compile different versions of your app anyway. For example, you probably won’t enable source maps for your production build. Webpack support this concept. http://webpack.github.io/docs/cli.html#development-shortcut-d

This technique is best suited for externalizing configuration that does not change throughout your build and deploy process. Turning console logging on/off, for example. Locally, you may want console logging to be on, but in all deployed environments you may not want to be doing that.

Use server-side templating to inject configuration at render time.

In other words, just throw some globals into the window object when the page is initially rendered. Here is an example using Pug (formerly known as Jade)

<!-- views/index.pug -->

doctype html
html(lang="en")
  head
    title My App
  body
    div(id="app")

    script(type="text/javascript").
      window.config = {API_URL: "#{API_URL}"};

    script(type="text/javascript" src="/app.js")

// server.js

const express = require('express')
const port = process.env.PORT || 8080
const app = express()

app.use(express.static(__dirname + '/public'))
app.use(express.static(__dirname + '/assets'))
app.set('view engine', 'pug')
app.get('/', function(req, res) {
  res.render('index', { API_URL: process.env.API_URL })
})

app.listen(port)

# running the server

export API_URL="http://dev.my-api.some-domain.com"
node server.js

// app.js

console.log('The API_URL is: ' + window.config.API_URL)

Two things here that are not so great. First, we are creating global state again. Second, we have to introduce a dependency on a templating engine. This means we can no longer just serve our client application as static assets. With option #1, we could have used something like Amazon S3 to deliver our client app, while now we have to maintain our own server in order to do the templating.

This approach has more of an old-school feel to it with the server-side templating and the global window variable.

It might be fine for you though if you are already using server-side templating. This is pretty much the way I did it in the last Rails app I worked on (back in 2014).

Conclusion

I think being able to injecting env properties at runtime is the way to go for large-scale production applications, as it gives you the most flexible options for local development and deployments. But, if you want a quick win for a side-project or small app, injecting at compile time is super easy. I probably wouldn’t use the server-side templating unless I was working on an application that was already doing that.

Harmonious Stylesheets and Inline Styles in React

Sat, 15 Oct 2016 14:28:00 GMT

Styling React components seems to be one of those things where there are a lot of solutions (libraries) emerging that try to solve the problem of how to use only inline styles in JavaScript sustainably. To me, it seems that this problem has been invented and can easily be avoided; and introducing a new library or tool is only adding to the complexity and cognitive overhead of your app. Many times it really is just better to stick with plain ol CSS stylesheets.

Both traditional stylesheets and inline styles have their place within React applications. By using them harmoniously you can have the best of both worlds.

What is style anyway?

The first thing to realize is that “style” is actually a few related concepts. I like to break it down into 2 main categories: presentational styles and behavioral styles.

Presentational styles describe the appearance of your application as a whole, and are rarely specific to a particular component. Instead, presentational styles define how elements look in relation to each other, and play the main role in the overall look-and-feel of the application. They are also always static. Base font styles, the color palette, and your grid system are all examples of presentational styling.

Behavioral styles describe how individual elements of your application look and behave in a given state. They are dynamic, requiring logic to determine. For example, if an element changes background colors when some state in the app is reached, then that is a behavioral style.

Prefer traditional stylesheets for presentational styling

For all the presentational styling needs of your application, stick with simple, traditional CSS stylesheets.

This will allow you to use modern CSS tools/languages such as Sass or LeSS and third party libraries such as Bootstrap or Bourbon.

Additionally, some CSS features such as media queries are at best painful and at worst impossible to do with inline styles alone.

Finally, for teams with non-developers such as Designers contributing to the styles, there is a high chance that they will be more comfortable and productive working with traditional stylesheets than inlined CSS inside of JavaScript files.

// Example of some presentational styles (Sass)

html {
  font-family: $font-family;
  @include base-font;
}

.button {
  padding: $spacing;
  background-color: $light-blue;
  color: $gray;

  &:hover {
    background-color: $dark-blue;
  }
}

Prefer inline styles for behavioral styling

Because behavioral styles are determined by application state, which will be held in JavaScript, keep them co-located with the component as inline styles.

This makes conditional state classes totally unnecessary. No more “is-complete” or “has-ratings” type of class names anymore.

Behavioral styles can and should be unit tested. This isn’t perfect though, even though your assertion that a particular style is applied passes, the component still may not render exactly the way you want it in the browser.

Here is a working example of a Nav component whose behavior is to slide out from the left side of the screen when opened. The dynamic state is sliderMenuVisible, which changes the left css property, triggering an animation. Notice the use of the nav class name for presentational styling concerns too.

// Nav.js

let styles = {
  transition: 'left 0.5s',
}

const Nav = ({ sliderMenuVisible, toggleNavSliderMenu }) => {
  if (sliderMenuVisible) {
    styles = { ...styles, left: 0 }
  }

  return (
    <nav className="nav" style={styles}>
      <ul>
        <li>
          <Link to="/browse" onClick={toggleNavSliderMenu}>
            Browse
          </Link>
        </li>
        <li>
          <Link to="/orders" onClick={toggleNavSliderMenu}>
            Past Orders
          </Link>
        </li>
      </ul>
    </nav>
  )
}

A Problem

The big problem here is that you cannot easily share code between CSS stylesheets and JavaScript inline styles. If you have defined a Sass variable $dark-red and also want to use that same color in a behavioral style, you would be forced to hard code the color value into the JavaScript too.

But, I have not seen this as a huge problem yet, and I don’t think this would be a hard problem to solve if it gets un-maintainable in your project.

Homogeneous Pipelines with Docker

Mon, 26 Sep 2016 09:55:17 GMT

Continuous Integration Build Pipelines are a dirty, nasty place. What usually starts out as a couple of simple tasks or bash scripts inevitably ends up as a heap of scripts, Gradle/Rake/Maven/(insert build tool here) tasks, and manually configured jobs held together with a thin layer of Elmer’s glue and Scotch tape.

Why? Partly because modern applications are complex; even simple stacks use multiple languages and tools. And partly because developers are lazy, and sometimes hesitant to “mess with the pipeline.” I don’t often see the amount of rigor in cleanliness applied to them as to other parts of the codebase, leading to unnecessarily complex and unfortunately tangled build pipelines.

So, my current team attacked this head-on using a great piece of technology: Docker! Using Docker as our sole interface to running things in our build pipeline, we sped it up, simplified it, and lived happily ever after.

To give credit where credit is due, the implementation of this pattern was spearheaded by my colleague Amber Houle.

Anatomy of a Pipeline

This is pretty much what our build pipeline looked like in the time before Docker. Notice the amount of variation in what is being invoked in each step! (All these steps could be wrapped up in bash scripts, but I’ve unravelled them here.)

“client” is a JS front-end application. “api” is a Java-based API.

# pipeline.yaml

build_client:
  - npm install       # Install JS dependencies
  - npm run webpack   # Compile to JS

build_api:
  - gradle build      # Compile Java

test_client:
  - npm install       # Install JS dependencies..... again
  - npm test          # Run JS tests

test_api:
  - systemctl start postgresql.service  # Start up Postgres
  - flyway migrate                      # Migrate the database using Flyway
  - gradle test                         # Run Java tests (unit and integration)
  - systemctl stop postgresql.service   # Stop Postgres

package_client:
  - npm install       # Install JS dependencies once again :(
  - npm run package   # Package up the JS into a .zip or .tar
  - <push JS code to artifact repository>

package_api:
  - gradle package    # Create an executable .jar
  - <push Java code to artifact repository>

deploy_to_qa:
  - <pull code from artifact repo and deploy>

Let’s first examine a build pipeline that you might find using any of the modern open-source distributed build and deploy tools such as Jenkins, Go.cd, or TravisCI. It’s broken down into a series of stages or jobs, which could be run sequentially or in parallel. Because these tools usually run as a master/agent architecture, the server will delegate the actual work of each stage to an available build agent.

Pipeline complexities

While this architecture is scalable and flexible, it creates complexities that you have to manage. Since each stage in your pipeline has a different job to do, all your agents must be configured to perform all needed actions. Some stages need a JavaScript runtime, some need Java, while others need a Postgres database. Traditionally, this calls for provisioning your agents with all the appropriate software ahead of time. And herein lies a dilemma. Manually provisioning might work fine if you only have 1 or 2 agents, but that quickly becomes tedious as the number of dependencies you have increases or the number of agents you need increases. Automated provisioning using Chef or Puppet is an option, but this creates one more piece of code to build, manage, test, and debug.

As each agent picks up a stage to run, it’s going to need some input, which is often just a copy of your source code at a specific revision. Because any agent could be picking up any job at any time, the sequence is usually 1) start with a clean workspace 2) checkout the code 3) install dependencies 4) do stuff. All these steps take time, especially installing dependencies. (There are 3 npm install commands in the pipeline shown above)

Docker as the Pipeline Interface

Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server.
https://www.docker.com/what-docker

What if we extend this statement to say that containers contain everything needed to run… and build and test? Instead of provisioning build agents with all the individual pieces of software and dependencies that our pipeline needs, let’s provision them with only the Docker Engine. Now, the sequence of steps for any stage becomes 1) Pull down a Docker image 2) Execute docker run https://docs.docker.com/engine/reference/run/.

To pull this off, we first need to create some Docker images with everything our application needs to build and test itself, which we specify with a Dockerfile.

# Dockerfile for Java API

FROM java:8

COPY build.gradle ./
COPY src ./src/

RUN gradle jar

ENTRYPOINT ["gradle"]

# Dockerfile for database migrations

FROM shouldbee/flyway

COPY ./src/main/resources/db/migration/*.sql ./sql/

ENTRYPOINT ["flyway"]

# Dockerfile for JS client

FROM node:6.4.0

COPY package.json ./
RUN npm install
COPY src ./src/
RUN npm run webpack

ENTRYPOINT [ "npm", "run" ]

And this is pretty much what our pipeline evolved into after transitioning to Docker…

# pipeline.yaml

build_client:
  - docker build -t client:${PIPELINE_ID} ./client  # Build Docker image
  - docker push client:${PIPELINE_ID}               # Push it to the container registry

build_api:
  - docker build -t api:${PIPELINE_ID} ./api
  - docker build -t migrations:${PIPELINE_ID} ./api

  - docker push api:${PIPELINE_ID}
  - docker push migrations:${PIPELINE_ID}

test_client:
  - docker pull client:${PIPELINE_ID}
  - docker run client:${PIPELINE_ID} test   # Run `npm run test` inside of the client container

test_api:
  - docker pull api:${PIPELINE_ID}
  - docker pull migrations:${PIPELINE_ID}

  - docker run migrations:${PIPELINE_ID} migrate  # Run `flyway migrate` inside of the migrations container
  - docker run api:${PIPELINE_ID} test            # Run `gradle test` inside of the api container

Complexities simplified

Overall, this had a number of positive effects on our build pipeline. First, speed: the time from pipeline start to ready to deploy to a QA environment dropped from ~12 minutes to ~4 minutes! This was largely due to no longer checking out the entire code base, installing dependencies, and re-compiling in each step of the pipeline. Second, simplicity. Configuring new build agents is now easy, as they only need Docker Engine. The single command interface is also cognitively simple. Each call to docker run [COMMAND] in the pipeline acts as a proxy to the task runner already in use in the codebase (e.g gradle [COMMAND] or npm run [COMMAND]), making it work just like local development without Docker.

The main possible issue I see with this is that the Docker container has a large surface area, which kind of goes against the advice from Docker to keep images as slim and trim as possible. I normally would not include all my test code into the deployable artifact that will eventually end up on my production server. Instead we have copied all the source and test code into the image. This could introduce dependency issues, such as security holes that may exist in libraries pulled in by test code. I have not observed this in practice though.

You will also notice that we push the Docker image to the registry right away. This ensures that the artifact that passes down the pipeline is exactly the same all the way through build, test, and deploy. But, it also means we are creating an artifact for un-verified code. What if the tests fail for a certain commit, but we have already created and pushed the Docker image? Do we leave it in the registry? Remove it? So far, we have just left them there, so not sure how this will play out in the long term.

Overall, this pattern has worked well for the team! Would love to hear your opinions and experiences with Docker in a build pipeline.

How your 'Mother' can help you 'Build' cleaner unit tests - Part III

Sun, 26 Jun 2016 21:28:58 GMT

We ended Part II with an inner Builder for Address, which forced us to revert to specifying seemingly arbitrary values in each test that needed to build an Address.

What I’ve usually seen at this point is pulling the Address objects into constants in each test class. As the test suite grows, the number of private static final Address WHATEVER_ADDRESS = ...; grows and grows. Doing that makes it hard to keep control over your test data, especially in a large codebase. A minor change to Address, such as a new invariant or an added field, could cause cascading changes to keep all those objects valid.

To avoid that, the next step is actually a very simple one, and boils down to centralizing the definition and creation of an Address with an Object Mother.

4) Introducing an Object Mother

An Object Mother is a type of Factory used to create example objects for testing. Let’s see it in action:

public class Address {
  // ... no change, still using inner Builder
}


// Test only class
public class AddressMother {
  public static Address.Builder address() {
    return new Address.Builder()
      .lineOne("123 Main St.")
      .lineTwo("")
      .city("Chicago")
      .state("IL");
  }

  public static Address.Builder hawaiianAddress() {
    return new Address.Builder()
      .lineOne("123 Hawaii St.")
      .lineTwo("")
      .city("Honolulu")
      .state("HI");
  }
}


import static AddressMother.address;

public class ShippingServiceTest {
  @Test
  public void shipsToTheAddress() {
    ShippingService service = new ShippingService();
    Address shippingAddress = address().build();

    Shipment shipment = service.shipTo(shippingAddress);

    assertTrue(shipment.wasSuccessful());
    assertEquals(shipment.getDeliveryAddress(), shippingAddress);
  }

  @Test
  public void cannotShipToHawaii() {
    ShippingService service = new ShippingService();

    Shipment shipment = service.shipTo(hawaiianAddress().build());

    assertFalse(shipment.wasSuccessful());
    assertEquals(shipment.getFailureReason(), "Cannot ship to Hawaii.");
  }
}

The Mother controls the values that constitute an example Address. Now, when a test needs an Address it asks the Mother for one. Your Mother could return fully initialized Address objects, but instead we have chosen to have it return Builders. This powerful variation allows a test to get a basic example object, and then modify it depending on what it is testing.

Give your Mother class the ability to build objects with various states as appropriate. In our case, we currently have the need for a basic, valid address, and a Hawaiian address. But, be wary of creating a different factory method for every example object you need. That may be a bit overkill. I’d say to only create factory methods for things that represent core use cases and not one for each edge case that you test. For example, if I wanted to test what happens when state is empty, I’d do address().state("").build() over creating an addresWithoutAState() factory method in the Mother.

The End

Whew, that’s it! Thanks for making it this far. If you skipped Parts I or II, I’d recommend going back to see the entire journey. We teased out a nice pattern for managing test data using the Builder pattern + Object Mothers. In doing so, we improved the design of our production code by making Address immutable and avoiding a sketchy constructor with a lot of parameters.

How your 'Mother' can help you 'Build' cleaner unit tests - Part II

Sat, 25 Jun 2016 14:20:26 GMT

In Part I, we created a separate Builder class in an attempt to move away from a constructor with too many parameters. In the process, the separate Builder ended up taking on the additional responsibility of supplying default values, blurring the line between a Builder and Factory.

Though a step in the right direction, it didn’t actually fix any of the Address constructor’s original problems, unfortunately. Oops. Other classes can still call it directly, meaning all of the risks of having a method with multiple parameters of the same type are still there, and we have another class in our system to maintain.

What we need is a strictly controlled way of creating Address objects that can be accessed by test code and prod code that is decoupled from the current constructor and it’s baggage.

3) Pulling the Builder inside Address

Let’s remodel the Builder as a static inner class within the Address class.

public class Address {
  private final String lineOne;
  private final String lineTwo;
  private final String city;
  private final String state;

  private Address() {
    // Prevent other objects from calling the constructor
  }

  // ... other methods omitted for brevity

  public static final class Builder {
    private String lineOne;
    private String lineTwo;
    private String city;
    private String state;

    public Builder lineOne(String lineOne) {
      this.lineOne = lineOne;
      return this;
    }

    public Builder lineTwo(String lineTwo) {
      this.lineTwo = lineTwo;
      return this;
    }

    // ... city() and state() builder methods omitted for brevity

    public Address build() {
      return new Address(this);
    }
  }

  private Address(Builder builder) {
    this.lineOne = builder.lineOne;
    this.lineTwo = builder.lineTwo;
    this.city = builder.city;
    this.state = builder.state;
  }
}


public class ShippingServiceTest {
  private Address.Builder shippingAddressBuilder;

  @Before
  public void setUp() {
    shippingAddressBuilder = new Address.Builder()
      .lineOne("123 Main St.")
      .lineTwo("")
      .city("Chicago")
      .state("IL");
  }

  @Test
  public void shipsToTheAddress() {
    ShippingService service = new ShippingService();
    Address shippingAddress = shippingAddressBuilder.build();

    Shipment shipment = service.shipTo(shippingAddress);

    assertTrue(shipment.wasSuccessful());
    assertEquals(shipment.getDeliveryAddress(), shippingAddress);
  }

  @Test
  public void cannotShipToHawaii() {
    ShippingService service = new ShippingService();
    Address shippingAddress = shippingAddressBuilder.state("HI").build();

    Shipment shipment = service.shipTo(shippingAddress);

    assertFalse(shipment.wasSuccessful());
    assertEquals(shipment.getFailureReason(), "Cannot ship to Hawaii.");
  }
}

Looking good Needs improvement

Looking good	Needs improvement
We have coupled the Builder to the `Address` class more tightly, which is good. They must change together. Also, we have finally gotten rid of the public all-args constructor with too many parameters! To take it even further, we removed all public constructors. The Private Constructor pattern tells collaborators that object construction is internal and explicitly controlled by the class itself. The Builder is now the only way to construct an `Address`.	However, the dual responsibilities of the separate Builder class have reared their ugly head. In order to prevent leaking test code (the default values) into our prod code (the inner Builder), we had to remove them altogether for now and go back to specifying them in the test class itself. This means we go back to the problem of confusing arbitrary values appearing in our tests.

We have coupled the Builder to the Address class more tightly, which is good. They must change together.

Also, we have finally gotten rid of the public all-args constructor with too many parameters!

To take it even further, we removed all public constructors. The Private Constructor pattern tells collaborators that object construction is internal and explicitly controlled by the class itself. The Builder is now the only way to construct an Address.

However, the dual responsibilities of the separate Builder class have reared their ugly head. In order to prevent leaking test code (the default values) into our prod code (the inner Builder), we had to remove them altogether for now and go back to specifying them in the test class itself.

This means we go back to the problem of confusing arbitrary values appearing in our tests.

Next up, Part III - where we truly divide the responsibilities for object construction into a Builder and a Factory, removing the arbitrary default values from the test itself.

How your 'Mother' can help you 'Build' cleaner unit tests - Part I

Thu, 23 Jun 2016 13:34:08 GMT

Over the past few months on my project, a pattern emerged for using Builders and Object Mothers to build-up objects for unit testing. We were able to keep our tests minimal and clean, decouple test code from production code, and solve some design issues such as constructors with too many parameters. In this multi-part series, I will take you through the evolution of the pattern over a series of refactorings.

The Scenario:

Note: This is contrived for simplicity

The basics: We are building a system that schedules shipments of widgets to a specified address.

The interesting bit: We have an Address class that we use quite often in our tests. By “use”, I mean instantiate one with some valid data to either exercise it or assert against it.

1) Initial Code

public class Address {
  private final String lineOne;
  private final String lineTwo;
  private final String city;
  private String state;

  public Address(String lineOne, String lineTwo, String city, String state) {
    this.lineOne = lineOne;
    this.lineTwo = lineTwo;
    this.city = city;
    this.state = state;
  }

  public void setState(String state) {
    this.state = state;
  }

  // ... other methods omitted for brevity
}


public class ShippingServiceTest {
  private Address shippingAddress;

  @Before
  public void setUp() {
    shippingAddress = new Address("123 Main St", "", "Chicago", "IL");
  }

  @Test
  public void shipsToTheAddress() {
    ShippingService service = new ShippingService();

    Shipment shipment = service.shipTo(shippingAddress);

    assertTrue(shipment.wasSuccessful());
    assertEquals(shipment.getDeliveryAddress(), shippingAddress);
  }

  @Test
  public void cannotShipToHawaii() {
    ShippingService service = new ShippingService();
    shippingAddress.setState("HI");

    Shipment shipment = service.shipTo(shippingAddress);

    assertFalse(shipment.wasSuccessful());
    assertEquals(shipment.getFailureReason(), "Cannot ship to Hawaii.");
  }
}

The two hot spots here are the Address constructor and the customization of the Address fields in the tests.

Let’s talk about the constructor:

It has too many parameters - four! (And you could imagine a real Address object having at least a couple more) And, multiple parameters in a row of the same type is even worse. If I accidentally swap state and city my program will still compile as both parameters are Strings, but it will probably fail at an unexpected time later.
The arbitrary values used to construct the Address do not reveal intention. Why “123 Main St”? Why “IL”? Could I change that to be any state? Which values are effecting the outcome of each test and which are completely arbitrary?
Optional parameters. lineTwo seems to be optional because of the empty String. Should we overload the constructor instead? Provide a setter?

And customizing the Address fields for each test:

Prefer immutable state. The setState method makes Address mutable. Before that, we had a nice, happy immutable object, as you can see by the presence of the final keyword on the other instance fields.
Because we were using the same Address object in many tests and we needed a different value for only one field, we added a setter instead of calling the constructor again. On the upside though, at least this test reveals its intention - that this test only cares about the state field.
Keep test-only code out of production code. It’s very likely that setState was added for this one test, and is not called by any production code. This is a smell that should be avoided.

2) Introducing a Builder

First, we attempt to give Address it’s immutability back and avoid adding telescoping constructors by introducing a Builder. The Builder allows us to separate the steps for constructing an object from the final representation of it.

public class Address {
  private final String lineOne;
  private final String lineTwo;
  private final String city;
  private final String state;

  public Address(String lineOne, String lineTwo, String city, String state) {
    this.lineOne = lineOne;
    this.lineTwo = lineTwo;
    this.city = city;
    this.state = state;
  }

  // ... other methods omitted for brevity
}


// Test only builder class
public class AddressBuilder {
  private String lineOne = "123 Main St";
  private String lineTwo = "";
  private String city = "Chicago";
  private String state = "IL";

  public AddressBuilder lineOne(String lineOne) {
    this.lineOne = lineOne;
    return this;
  }

  public AddressBuilder lineTwo(String lineTwo) {
    this.lineTwo = lineTwo;
    return this;
  }

  // ... other methods omitted for brevity

  public Address build() {
    return new Address(lineOne, lineTwo, city, state);
  }
}


public class ShippingServiceTest {
  private Address shippingAddress;

  @Before
  public void setUp() {
    shippingAddress = new AddressBuilder().build();
  }

  @Test
  public void shipsToTheAddress() {
    ShippingService service = new ShippingService();

    Shipment shipment = service.shipTo(shippingAddress);

    assertTrue(shipment.wasSuccessful());
    assertEquals(shipment.getDeliveryAddress(), shippingAddress);
  }

  @Test
  public void cannotShipToHawaii() {
    ShippingService service = new ShippingService();
    shippingAddress = new AddressBuilder().state("HI").build();

    Shipment shipment = service.shipTo(shippingAddress);

    assertFalse(shipment.wasSuccessful());
    assertEquals(shipment.getFailureReason(), "Cannot ship to Hawaii.");
  }
}

Looking good Needs improvement

Looking good	Needs improvement
The Builder lets tests construct `Address` instances without invoking the public constructor, reducing the coupling to it. The addition of default values to the Builder pulls that arbitrary data out of the tests themselves, which further helps to highlight any tests that need a specific value, and without needing a setter. `Address` is immutable. :)	This is yet another class to maintain. By making the Builder a separate object, we still need `Address`'s public all-args constructor. Any other test could easily bypass the builder. And, because the constructor is still there, all the problems from before remain. Additionally, our Builder actually has two responsibilities. First is how to construct the `Address`, as in, which parameters to pass into the constructor. Second, what to construct it with, because the Builder has default values.

The Builder lets tests construct Address instances without invoking the public constructor, reducing the coupling to it.

The addition of default values to the Builder pulls that arbitrary data out of the tests themselves, which further helps to highlight any tests that need a specific value, and without needing a setter.

Address is immutable. :)

This is yet another class to maintain.

By making the Builder a separate object, we still need Address's public all-args constructor. Any other test could easily bypass the builder. And, because the constructor is still there, all the problems from before remain.

Additionally, our Builder actually has two responsibilities. First is how to construct the Address, as in, which parameters to pass into the constructor. Second, what to construct it with, because the Builder has default values.

Part II will look at moving the Builder into a static inner class to fix the constructor with too many parameters issue.

YAGNI a'int a Design Principle

Fri, 12 Feb 2016 13:21:24 GMT

You Ain’t Gonna Need It is an XP practice that encourages developers to focus on the requirements at hand right now. It warns against the tendency to over-engineer by developing features based on what you think you will need later. Easy in theory. Hard in practice.

YAGNI Blinders

I’ve noticed that developers, especially those just starting out, tend to apply YAGNI narrow-sightedly. I often see it being strictly adhered to at face-value only, by developers always choosing to do what seems to be the most obvious and straightforward thing at the time, without putting any thought into the design implications of their choices. When suggesting other options that seem to be less simple on the surface, they YAGNI argument gets used. “Oh, that’s YAGNI.”

YAGNI encourages simplicity. But many developers read simple as easy, which can lead to design problems with their code that easily have been avoided. An easy solution doesn’t require any thought or skill. A simple solution involves elegance, is purposeful, and is built with extensibility and evolvability in mind. (Rich Hickey, the creator of Clojure, gave a wonderful talk along these same lines in 2011. Simple Made Easy)

As an example of YAGNI Blinders, take internationalization (i18n) of a web-app. Usually, the first requirement is to support only a single language, with support for other languages pretty far down the line. A classic YAGNI solution here: hard coding everything in the HTML files. Done. (I’ll come back to this soon….)

See the Light!

The YAGNI argument breaks down because it does not take critical design characteristics such as readability, flexibility, and loose coupling into account. A purely YAGNI-based solution does not necessarily imply those things. By sticking to a YAGNI-only solution, you make it harder to quickly move forward later, because extensive refactoring or rewriting would be needed first. By considering simplicity along with many other design principles, you will end up with better factored code.

The most obvious choice is not always the right one. YAGNI does not mean to plow headfirst into the first solution you think of. Good design trumps. But figuring out how to satisfy the requirements at hand, while building a simple, flexible solution without gold-plating, over-engineering, or solving for unknown future requirements is hard! Good. Glad our job isn’t boring. :)

Back to i18n, instead of hard coding all the strings in the HTML, we can introduce a simple lookup mechanism. We store all our strings in a JSON file, and create a simple module that loads that file into a map structure, so that strings can be looked up by a key.

Yes, this is not the absolute easiest solution. However, it is simple, and a better design. We have better separation of concerns. We have pushed configuration (the strings) out of code, allowing them to change independently as they may change at a different rate than the application code. We could extend the lookup mechanism without changing the strings or keys. Event If we never get a requirement to add another language, this is still a better design, and we did not have to build a complex architecture involving locales, multiple languages, unicode character support, or any of the other myriad of problems that come with i18n. Not easy, but simple.

You always need good design. If you find yourself with a pair of YAGNI blinders on, ignoring design and code quality for purely YAGNI arguments, take a step back and remember that YAGNI ain’t a design principle.

You're a Champion

Sat, 23 Jan 2016 14:04:06 GMT

Recently, my team embarked on a strategy/experiment designed to distribute the effort of driving features from inception through completion more evenly across the team. We noticed that both myself (Tech Lead) and our BA were overloaded. Too many features. Too little time. What’s worse, our developers were feeling out of the loop most of the time. They wanted more insight into what goes on before a story gets to development and more say in the direction of the team.

Thus, Feature Champions were born.

A Feature Champion is the champion of a feature throughout its entire lifecycle, naturally becoming the knowledge expert in that feature.

A Champion can be anybody on the team, though we chose to limit our Champions to the developers and QAs. It may sound like a lot of pressure, but that’s why the word champion is important. Champion != owner. Championing is not meant to introduce silos of knowledge or eliminate roles. Championing does not leave the champion alone to do the entire feature in isolation. Above all else, the Champion is a collaborative role that works closely with other team members and stakeholders.

So, what does a feature champion actually do?

engages with team/feature stakeholders to gather requirements and priority
takes guidance from BAs/QAs to help with business and quality analysis
works with the PM/IM to track the progress and scope of the feature
participates in architecture and technical discussions with the tech lead or architects
presents the feature to the team as the knowledge expert
acts as a touch person for questions about the feature from other teams

Notice the big thing that is not in this list: Does all the development on the feature themselves. No! You are a Feature Champion, not a Feature Owner. You guide the feature throughout its wonderful life, but rely heavily on support from the entire team along the way.

The Road Ahead

Some developers have taken to this exceptionally well - researching, scheduling meetings with stakeholders, writing stories, and asking tons of questions. Others have struggled with the less structured role. Championing some features involves much more than coming into work and writing code all day. Learning new skills is always difficult. It has forced us to prioritize our work. Now, a dev might be wearing many hats throughout the day, coding on a story for awhile, then doing analysis for a feature for awhile.

To outsiders looking in, it may seem like the team’s velocity would take a hit. While we have not done this long enough to tell for sure, I don’t think this will happen in the long term. I see the team ending up with more shared knowledge about the product, and more understanding of other roles. So, when the BA or Tech Lead win the lottery, the rest of the team already has the experience to step up and fill that gap.

Finally, I see Feature Championing as being a great way to build up less experienced devs in a safe way. They are learning what goes into analyzing a feature and interacting with non-technical stakeholders. They are thinking at a higher level technically about architecture and system design.

Database Migrations Best Practices

Sat, 15 Aug 2015 15:53:23 GMT

Only once have I worked on a project that did not interact directly with a relational database. (Awww, those were the days) In every other case, I used database migrations to apply small changes to a database over time. Using migrations provides an incremental path towards building the current state of a database. In my opinion it’s the only way to do it.

Database migrations are nothing new, and over the years I have come to follow a few best practices:

1 Migration - 1 Change - 1 Commit

Each migration should execute a single unit of work on the database. This comes from the Agile software development principle of introducing small, incremental changes. Small migrations are easier to read and understand.

For example, if I need to create a customers table and accounts table, I would create one migration to create the customers table and a second migration to create the accounts table. Two migrations. Two commits.

By keeping each migration contained in its own commit, it is easy for automated build systems to apply them 1-by-1 in later environments. And if a migration turns out to be malformed, you make it easy to revert just that change if necessary.

I always try to commit a migration with no code changes attached. This isolation ensures that the migration is fully compatible with the application code both before and after it. Your Database Administrators will also thank you for making it easier to notice the migration when its all by itself.

Never modify a previously committed migration

(This is the one that seems to be unintuitive for less experienced developers.)

Once a commit has left my local machine, it is effectively locked! If I notice afterwards that there is a typo in a column name or some other mistake, I create another migration that fixes it rather than modifying the bad one.

Why? Imagine this interaction between 2 developers, Patrick and Molly:

Patrick	Molly
1. Patrick commits migration #8, which creates a table with a column "product_typ" (misspelled it, oops)
	2. Molly pulls new code and applies migrations, which gives her the new table with the misspelled column name
3. Patrick realizes his mistake, so he rolls back migration #8 locally, dropping the table. He then modifies the migration to fix the typo, then applies migration #8 again.
4. Patrick pushes his changes.
	5. Molly pulls new code and applies migrations again. But this time no change is made to her database because she already applied migration #8 and no new migrations were added. She is stuck with the typo. :(
6. Patrick's and Molly's database schemas are now out of sync.

The only way Molly can fix her database now is to drop it and start over. Depending on the migration and what was changed, she may not be able to rollback for a quick fix. Want a scarier scenario? What if Molly was actually the production database… Oops indeed.

Save yourself the pain. Just don’t do it.

Avoid using your application code (such as models) in migrations

Some migration tools allow you to use application code in your migration files, especially if you are using the ActiveRecord pattern (I’m looking at you Rails). The use case for this could be an easy way to do some data manipulation, or inserting seed data.

I avoid doing this because you cannot guarantee that your code will keep the same API forever. Even the first migration could be run at any point in the future (say a new dev joins and runs them all for the first time), you can’t assume those same methods could be called. (And I don’t change previously run migrations)

P.S. Rails 4 no longer even mentions this as an option. The link above from Rails Guides actually warns against this practice too. :)

Keep migrations abstract

Try not to use database-specific operations. This couples you to that database technology. Most migration tools/frameworks attempt to abstract the database away for you, so by using database-specific concepts or operations, you break that abstraction layer.

However, this rule is often broken since each database speaks its own version of SQL.

Think about the data

I often see developers overlooking the data stored in their databases. Migrations don’t just effect the schema, they also effect the stored data - a thats the important part! This happens a lot because our local development databases may not have very production-like data.

Example: Consider a migration that applies a not null constraint to a column.

Local DB	Production DB
employees id \| name \| phone_number --------------------------- 1 \| foo \| 5555555555 2 \| l33t haxor \| 1234567890	employees id \| name \| phone_number --------------------------- 1 \| Patrick \| 5554567890 2 \| Molly \| null

ALTER TABLE employees MODIFY phone_number VARCHAR(255) NOT NULL;

This migration would be fine in development, but that migration would fail in prod. A better migration would actually be 3 steps:

Insert a value into every row where phone_number is currently null
Add a default value for future inserts (if applicable)
Introduce the not null constraint.

Consider changing a column’s data type from a floating point precision to an integer. Maybe you are converting your money column to be stored as cents so you can avoid floating point mathematical errors. What would happen to the precision when the migration is applied? Would the database truncate it? Round it? Something else?

A more pragmatic approach may actually require many steps: introduce another column amount_in_cents; move all the current data to the new column and perform the format change; drop the amount column; finally, rename the amount_in_cents column to amount.

These are only two scenarios I’ve seen lately, but there are many more.

Moar

Check out Database Refactoring for a lot of good database refactoring patterns.

Hackathons for the Enterprise

Sun, 26 Jul 2015 10:30:04 GMT

A few months ago I helped to plan and execute a Hackathon for my project team. Though it was not the first team Hackathon I have participated in, it was the most successful. Everyone on the team had a great time and learned a lot. Some of the hacks have even been turned into production solutions for the product or team!

I decided to take this experience and share it with a wide audience by contributing to the ThoughtWorks Insights blog. This is extremely exciting for me, as its my first time to be featured alongside so many of my brilliant colleagues! :)

Check out my article here.

Watching the Watchers in AngularJS

Tue, 07 Apr 2015 08:43:55 GMT

I have been doing quite a bit of AngularJS performance tweaking on my latest project. It is no secret that Angular performance can quickly degrade due to the implementation of its digest cycle. I’ll avoid Angular bashing here because this is well-documented and blogged about to death (Google it). You’re welcome.

Instead, I want to share a few learnings that I have picked up while attacking the performance problems with having too many bindings and watchers in any Angular application.

Should I even be worried?

The first step is actually knowing if you have a problem. Eventually, on my project, we could feel the responsiveness of our application start to slip. Rendering seemed to slow down. Interactions started to feel sluggish. You would click a toggle, it would wait a tick, and then change states. Things like that. Even if you aren’t noticing anything this drastic, its still a good idea to get some insight and follow best practices though.

In order to help pinpoint the hotspots in your Angular app, I developed this Chrome extension to count how many watchers different sections of your app contain.

Bind once is your friend

This one is the easiest and has a big ROI. When using the normal binding syntax {{ <expression> }}, Angular sets up a two-way binding so that the view can be automatically upated when the underlying data changes. However, there are probably some static data in your app, which makes the two-way binding both redundant and costly because Angular re-evaluates that expression during every digest cycle to update your view with any changes. The bind once syntax eliminates that. Quick win.

{{ ::item.name }}

Note: only available in Angular 1.3 and later

Avoid excess use of ng-show/ng-hide

These directives are easy to abuse. When you have multiple elements that need to be hidden or shown based on similar conditions, you can often combine multiple ng-show/ng-hides into a single ng-class and sprinkle in some targetted CSS rules to achieve the same behavior with fewer watchers.

So this, which contains 3 ng-show/ng-hide watchers…

<div>
  <div>
    <span ng-hide="item.isSoldOut()">Price: $49.99</span>
    <span ng-show="item.isSoldOut()">Not in Stock</span>
  </div>
  <div>{{ ::item.name }}</div>
  <div ng-hide="item.isSoldOut()">
    Ships within 3-5 business days
  </div>
</div>

Can be turned into this, which contains 1 watcher - the ng-class:

.item .show-when-sold-out {
  display: none;
}

.item.sold-out .show-when-sold-out {
  display: block;
}
.item.sold-out .hide-when-sold-out {
  display: none;
}

<div class="item" ng-class="{sold-out: item.isSoldOut()}">
  <div>
    <span class="hide-when-sold-out">Price: $49.99</span>
    <span class="show-when-sold-out">Not in Stock</span>
  </div>
  <div>{{ ::item.name }}</div>
  <div class="hide-when-sold-out">
    Ships within 3-5 business days
  </div>
</div>

Beware the ng-repeat

A = How many watchers does each element of your ng-repeat contain?
B = How many elements will you be iterating over?

if(A * B > 3000) { console.log("Houston we have a problem"); }

On my current project, we were blindly using ng-repeat on elements that contained over 50 watchers each. This was fine for some users who only had a few items in the list. But when we hit some users who loaded over 400 elements into that list, (50 * 400 = 20,000), our digest cycle slowed to a crawl and we even crashed the browser sometimes. (Especially mobile browesers.) Oops.

Besides using the other tips listed on this page to reduce the overall number of watchers, my advice is to think about the user experience of your app. Do you really need to render all those elements in one big list? Enter pagination. Enter filtering and searching.

We went with a hand-rolled pagination solution because the available plugins didn’t quite fit our needs.

Lazy evaluation of DOM elements

Take this example:

<div class="item">
  <div class="item-header">
    <span>{{{ ::item.name }}</span><span ng-click="item.expand()">+</span>
  </div>
  <div class="item-info" ng-show="item.isExpanded()">
    <p>{{ item.description }}</p>
    <form ng-submit="item.addComment(comment)">
      <input type="text" ng-model="comment" />
      <input type="submit" />
    </form>
    <!-- More bindings and watchers -->
  </div>
</div>

Even though the elements in div.item-info are hidden, the watchers and bindings on those elements will still be evaluated by Angular during the digest cycles. There is actually no point in compiling and linking all the DOM elements in the hidden div.item-info section until the user actually clicks the expansion trigger. We found that we could get a huge performance boost by taking advantage of templates and changing the structure of our HTML a bit. So when the user clicks on the expansion trigger, we grab the appropriate template, compile and link it with the data, and insert that into the DOM!

<div class="item">
  <div class="item-header">
    <span>{{ ::item.name }}</span><span ng-click="item.expand()">+</span>
  </div>
  <div class="item-info"></div>
</div>

<script type="text/ng-template" id="expandedItem.html">
  <p>{{ item.description }}</p>
  <form ng-submit="item.addComment(comment)">
    <input type="text" ng-model="comment" />
    <input type="submit" />
  </form>
  <!-- More bindings and watchers -->
</script>

The one caveat here is that once the user has expanded the item, all the watchers and bindings from the expanded item are now part of the digest cycle. The more elements you expand, the more watchers you end up with. This worked on my application because we expect the user to only expand a few elements out of a long list - a big savings. If you expect the user to quickly show all the elements you were initially hiding, this may not help much. In any case, it should at least help with initial page load time because the ng-templates are not compiled until you explicity tell Angular to.

Resources

I read a lot of blog articles and documentation while messing around with all of this. One that I found particularly useful was Ng Nuggets. Thanks!

Easing into Rubocop with git

Wed, 25 Feb 2015 20:08:31 GMT

Dropping Rubocop into an existing codebase

We (my team) recently introduced rubocop to a Rails app and a Sinatra app to encourage (aka enforce) us to follow established Ruby style and semantics. This was my first experience with dropping a static code analyzer into an established codebase. Just for reference, our Rails app is not huge - 16 controllers, 25 models, 10 services, and a smattering of other files.

Even so, the first run of bundle exec rubocop on the entire application revealed about 2,500 warnings. Whomp whomp. Thankfully, Rubocop comes with a handy-dandy autocorrect feature. A quick bundle exec rubocop --auto-correct fixed almost half of them. Good, but not quite there yet.

Incremental Rubocoping

We wanted to Rubocop to execute as part of our pre-commit task, and we wanted it to fail the task if it produced warnings or errors. So, our options were:

Have someone sit down with a case or two of Red Bull and don’t stop until everything is fixed.
Use the “Automatically Generated Configuration” from Rubocop, which generates a config file for you from all the warnings with all those cops turned off, letting you choose when to enable them.
Configure Rubocop to only run on a subset of files or directories. Fix all those warnings before adding more files, rinse and repeat until the entire app is added.

We went with… kind of 3. We liked the idea of incrementally rubocop-ing our app as we worked on it. So our solution is based on the Boy Scout Rule - strive to always leave any code you touch in a better state than when you found it. Applying that to Rubocop means every time you commit, Rubocop gets run ONLY on the files you have touched in that commit. Over time, we should cover more and more of the app until eventually we can run Rubocop on the entire app with every commit! Boom!

The one caveat is that you must remember to run the task BEFORE you do git commit in order for the task to pick up your changed files. I’m sure we could put in some more effort to account for that, but haven’t done so yet.

The Gist of it

embed:easing-into-rubocop-with-git/git_rubocop.ruby

How to Support Apprentices on an Agile Development Team

Mon, 26 Jan 2015 19:39:06 GMT

Recently, my team was tasked with figuring out how to introduce what we chose to call “Apprentice Developers” into the team in order to help on board new talent into the company. How would our day-to-day interactions and development practices have to change in order for less experienced team members to grow while being productive? I was pretty excited about this initiative, as I enjoy mentoring others to become better software developers and being part of effective, high-performing teams.

Our definition for Apprentice Developer:
Entry-level Dev. Probably none or less than a year of full-time professional experience in software development.

First some background, in the past few months we have been supporting a couple Apprentice-level developers with various amounts of success. Until now, the development team had been a bit loose in adhering to XP practices, such as pairing. Sometimes we would, sometimes we wouldn’t. Sometimes we would switch pairs every day or so, but there was no real cadence to it. Being a small agile team (7 devs), this kind of thing was fine for awhile, but we could easily see that adding more less-experienced devs would require a tad more structured of an environment.

Our first step was to take on honest look at our dev team culture and how everyone was responding to it. We discussed the worries that the team had and we solicited feedback from the current Apprentice developers centered around how the team had supported them, and what areas there might be for improvement.

We came to the conclusion that we would identify five new development practices that promote a more supportive and safe environment for Apprentice-level developers. So we carved out about two hours to come up with them, which I facilitated. To start things off, we narrowed our focus just a bit by creating a few categories that we thought would have challenges from the initial group discussion notes and feedback. From here, we took about five minutes to do some divergent thinking and come up with as many ideas as we could for addressing anything in any of the categories. After reviewing, clarifying, and grouping similar ideas, we each used five votes to start converging on the ones that we wanted to take further. It looked like this.

Next, we discussed the ideas one-by-one in descending order of votes, with the outcome of each discussion being a tangible, immediately-implementable practice for the team. Here is what we came up with:

As facilitator, the most obvious feedback for myself is that I was not an impartial outside facilitator. I had my own ideas and biases. Though I consciously tried not to let them interfere with any discussions, I’m sure that happened a little bit. That is probably unavoidable and a big reason why an outside facilitator for exercises such as this one is preferable.

I also could have watched the time a little better. It took over 2 hours, which I feel was way long. We got off track quite a few times. And the discussions after voting may have been a bit too open ended. Looking back, having a set time box for each post-vote discussion may have provided just enough time pressure to spur the group to create something tangible a bit quicker.

At the end of it all though, the team agreed upon five new practices to enact right away. The underlying goal was to help support new Apprentice developers, but I think the entire team, regardless of experience or skill level, will benefit from them. I’m looking forward to seeing how it goes and how the team grows!

Testing JavaScript Web Workers with Jasmine

Fri, 29 Aug 2014 15:48:00 GMT

JavaScript Web Workers have been around for awhile now, but I had not needed them until recently. Without going into too much domain specific info about the actual use case, I decided to go with Web Workers to handle map reduce style statistic calculations on a data set in the browser.

I was stoked to find the Web Worker API small and straightforward, making it super easy to get up and running. The only real speed bump while getting started was the lack of support in older browsers (IE8 and IE9 you ruin everything). However, turns out there is already a polyfill that works great. :) https://code.google.com/p/ie-web-worker/

I am a big proponent of testing my code. After some Googling, I didn’t find anything talking about testing JavaScript Web Workers, hence, this article.

First Attempt:

A long running background worker would be difficult to properly unit test, but my case was a bit simpler. I was posting data to the worker and letting it spit a result back out. I decided to just try the simplest Jasmine test first:

Note: This is just an example with a similar structure as my actual app.

// sum_foo.js

onmessage = function(e) {
  var array = e.data

  var sum = array.reduce(function(sum, element) {
    return sum + element.foo
  }, 0)

  postMessage(sum)
}

// sum_foo_spec.js

it('sums the values of foo', function() {
  var worker = new Worker('sum_foo.js')
  worker.onmessage = function(result) {
    expect(result.data).toBe(6)
  }

  worker.postMessage([{ foo: 1 }, { foo: 2 }, { foo: 3 }])
})

Surprising to me, this didn’t work! :( The test seemed to pass, but there was a Jasmine error.

Uncaught TypeError: Cannot read property 'expect' of null

What seemed to be going on is that, since this is an asynchronous test, by the time the execution of the test reached the expectation, the jasmine environment was no longer valid or able to perform the assertion.

UPDATE:

So in the process of writing this I discovered a more correct solution to my problem, which I have included here. But I decided to keep around the whole post because of the Rails intricacies and my overall problem solving thought process.

Turns out that Jasmine already has support for these type of asynchronous operations with the use of a done() function, that Jasmine will use to know when an asynchronous test has finished. http://jasmine.github.io/2.0/introduction.html#section-Asynchronous_Support

// sum_foo.js_spec

it('sums the values of foo', function(done) {
  var worker = new Worker('sum_foo.js')
  worker.onmessage = function(result) {
    expect(result.data).toBe(6)
    done()
  }

  worker.postMessage([{ foo: 1 }, { foo: 2 }, { foo: 3 }])
})

This is the solution I will be going with, but if you keep reading you will see something I came up with that uses promises to place the assertion AFTER “postMessage,” which I find easier to read and reason about when doing asynchronous tests.

Lesson learned here: always read the documentation fully and upgrade if you can first.

Second Attempt:

Time to be clever. Since my goal was to test the Web Worker code itself, I decided to reverse engineer the Web Worker API. I realized that the Worker was making an XMLHttpRequest to grab the script and then executing the code in its own context, so I took a similar strategy:

// sum_foo_spec.js

it('sums the values of foo', function() {
  var http = new XMLHttpRequest()
  http.open('GET', 'sum_foo.js', false)
  http.send()

  var workerCode = http.responseText

  // This will define the worker's "onmessage" function in the context of this test
  eval(workerCode)

  // Callback when the worker has done its work
  function postMessage(result) {
    expect(result).toBe(6)
  }

  // Execute the action under test
  onmessage({ data: [{ foo: 1 }, { foo: 2 }, { foo: 3 }] })
})

Success!

Improvements:

Now that I had a working solution, I had to write more tests for more workers (so far my app has 14 workers and maybe more to come), which means reusability. I wanted to extract away all the hairiness of requesting the worker script and evaling it into the current context. I also don’t like writing the expectation before the action of the test, so I turned to promises to help out.

Note: using jQuery’s Deferred here as my promise library because I already have jQuery in the project.

// worker_helper.js

var getWorker = function(path) {
  http = new XMLHttpRequest()
  http.open('GET', path, false)
  http.send()

  return http.responseText
}

var workerTester = function(workerCode) {
  var deferred = $.Deferred()

  // Define onmessage from the worker
  eval(workerCode)

  // The worker will call this method with the post-back data
  function postMessage(data) {
    deferred.resolve(data)
  }

  var thenAssertOn = function(assertion) {
    deferred.promise().then(function(data) {
      assertion(data)
    })
  }

  var sendMessage = function(data) {
    // Call into the worker code
    onmessage({ data: data })

    return { thenAssertOn: thenAssertOn }
  }

  return { sendMessage: sendMessage }
}

// sum_foo_spec.js

var workerCode = getWorker('sum_foo.js')

it('sums the values of foo', function() {
  workerTester(workerCode)
    .sendMessage([{ foo: 1 }, { foo: 2 }, { foo: 3 }])
    .thenAssertOn(function(sum) {
      expect(sum).toBe(6)
    })
})

Bingo! I was pretty happy with the final solution. It made testing of the rest of the workers trivial. And it works both in the browser and in a headless environment such as phantomjs.

Rails Setup:

I am using Rails 4 on the backend for this, which actually took a bit of time to get everything set up to work correctly in a Rails pipeline. Here is what I’ve done.

All workers are in a separate folder: “app/assets/javascripts/workers”
“application.js” does NOT require the workers. As I alluded to, when instantiating a worker with new Worker(‘script_name.js’), an AJAX request is made to the server to fetch the resource, so compiling it into application.js isn’t necessary
Add all workers to the precompile array: config.assets.precompile += Dir.chdir(File.join(Rails.root, 'app/assets/javascripts')) { Dir['workers/*.js'] }
Instantiate workers using inline JavaScript in application.html: new Worker('#{javascript_path("workers/script_name.js")}'); Notice the use of `javascriptpath`. The workers are being precompiled by the asset pipeline and will need the MD5 checksum._

The final issue was with jasmine_rails. Running rake spec:javascript worked fine, but when running RAILS_ENV=test rake spec:javascript, the worker scripts were not able to be fetched and thus a lot of tests failed. When jasmine rails runs, it copies all the files it needs into its own temp directory (tmp/jasmine by default). I ended up figuring out that running the jasmine specs in the TEST environment causes the src and spec files you specified in your jasmine.yml to be concatenated into a single jasmine-specs.js file, copied into tmp/jasmine, and included in the jasmine runner.html file. This meant the workers were not available to be fetched via AJAX. The solution I found is to use a custom spec runner layout file that manually includes the workers. This causes them to be copied into tmp/jasmine along with the concatenated jasmine-specs.js file, and available for fetching by the Web Worker.

// app/views/layouts/jasmine_rails/spec_runner.html.haml

!!!
%html
  %head
    %meta{content: 'text/html;charset=UTF-8', 'http-equiv' => 'Content-Type'}
    %title
      Jasmine Specs

    = stylesheet_link_tag *jasmine_css_files

  %body
    #jasmine_content
    = yield

    = javascript_include_tag *jasmine_js_files
    = javascript_include_tag *Dir.chdir(File.join(Rails.root, 'app/assets/javascripts')) { Dir['workers/*.js'] }

(https://github.com/searls/jasmine-rails#custom-helpers)

So a few hoops to jump through, but now I’m very happy with the Web Workers and the testing strategy I arrived at.

GOTO Chicago 2014 Experience Report

Mon, 26 May 2014 13:31:00 GMT

I had the privilege of attending Goto;con 2014 in Chicago. It was smaller than I anticipated, spread over just a few conference rooms in a small corner of the Drake Hotel, with no more than 400 attendees. This gave the conference a fairly informal, casual vibe. The talks were spread over a variety of tracks, from distributed systems to UX to functional programming languages, which kept things fresh over the 2 day conference.

Here I will capture summaries and my key takeaways from some of the sessions I attended (which represents only a small fraction of the 40+ presentations).

Camille Fournier: So You Want to Rewrite That…

Head of Engineering, Rent the Runway

Camille extrapolated some lessons from a successful rewrite of PHP Drupal system to a more modular platform built on a Ruby frontend with supporting Java microservices. Two of her main suggestions in order to make a rewrite sustainable were to 1. change as little as possible, and 2. to change only one thing at a time. Keep it small and incremental. She went on to discuss how to sell a rewrite to the business by making it clear what “done” will look like for the stakeholders. She also mentioned potential pitfalls, such as a version 2 that tries to do everything because of too many additional features being added during the rewrite. There is also the worry of over engineering. And if the team does not have a good idea of what done means, you could end up never really getting there.

Her overall message was that a rewrite can be successful, and to make that happen you should make incremental changes that provide value.

Reid Draper: Forty Years of Pretending

Profesional Erlang Developer, Works on Riak

Reid spoke about the recent resurgence of RPC-style architectures in JavaScript heavy web apps. At first I did not quite buy it, was he saying we are going back towards RPC? No way! But he ended up connecting the dots in the end.

He reminded us of some of the glaring problems with RPC:

It is difficult to keep consistent code on every node. If you need to update code, then you have to update it everywhere. Think of JS on clients as the nodes. You don’t control which version of the JS the client has. With JS on browsers, how to you deliver updated code? Do you force users to reload the page?
How do you handle failure? HTTP is inherently unstable.
How do you serialize complex objects such as DB connection objects?
Streaming

He went on to talk about how, since we can write JS on the front end and back end now, we may be tempted to use this RPC-style architecture and run the same code on both ends. He briefly noted a few JS-RPC libraries and advised against using them, since we already know the shortcomings of this approach.

He then walked through the Fallacies of Distributed Computing:

The network is reliable
Latency is zero
Bandwidth is infinite
The network is secure
The network topology does not change
There is a single network admin
Transport cost is 0
The network is homogeneous

I have been kind of unconsciously aware of these while developing web and mobile applications, but it was awesome to make them really explicit. He used them to stress the difficulties in developing JS web apps and brought up a great final point.

“Even a simple web app is a distributed system”

Michael Nygard: Five Years of DevOps: Where are we Now?

Author of “Release It!”

It was pretty great to hear a talk about DevOps from Michael Nygard. He started with the basics and growth of DevOps, including CAMS (Culture, Automation, Measurement, Sharing). He made a good point by applying Donella Meadows’ “Leverage Points in a System” to DevOps, and how the best place to affect an organization with DevOps comes at the information flow level and goals of the organization. Eventually, he outlined that DevOps over the past few years has come to mean:

CAMS + human factors + Agile values + Continuous Delivery = DevOps

I really enjoyed his scorecard of where we are currently at with DevOps:

Category	Score
Deployments	A	Most deployment tech right now targets a single machine, not a cluster
Provisioning	B
Logging	A+
Monitoring	A+	Graph all the the things!
Anomaly Detection	C
System Comprehension	D+	We are not very good at connecting a machine being down to the implications on the rest of the system and to the bottom line of the organization

Going forward, some more things to expect in this space:

More focus on anti-fragility - systems that improve from randomness. He used the Symian Army from Netflix as a current example.
A culture shift to viewing development infrastructure as mission critical. “Development is Production”

Finally, some cautions: Michael thinks DevOps is at the top of the “innovative technology adoption curve” right now.

There is a market penetration and DevOps is the most popular one at the party. He warned against trying to emulate practices without adopting the DevOps culture though. There is no such thing as DevOps-in-a-box. Having a separate DevOps team is also an anti-pattern.

Martin Odersky: Scala - The Simple Parts

Creator of Scala, Co-founder and Chairman of Typesafe

Martin basically wanted to address some bad rap Scala has gotten lately about it being overly complex and huge (I think so too). He called it a growable and modular language. It is growable in the sense that it is easy to build a DSL on top to expand its capabilities (such as Spark, Chisel, Akka, etc). It is a language for growth - you can start fast, with a 1-liner program. You can experiment fast. Then, it can grow very huge by using a combination of OO and functional programming. Large systems need both. This leads to also viewing Scala as a modular language. Modules are essential pieces of both OO and functional languages. With modular programming, your focus in on combining modules to do interesting things. Scala give you the power to choose how to combine modules, and which programming paradigm to use.

Martin was trying to stress that Scala has some simple building blocks (remember that simple != easy):

Everything is an expression
Scopes allow you to nest everything (functions inside of functions)
Patterns and case classes
Recursion and more importantly, tail recursion
Function values
Collections. Think transformations, not CRUD operations
vars. Combine a little bit of mutable state

Overall, to me the talk actually had the opposite effect, it showed me how little I understand of some of the principles on which Scala is built. During the last half of the talk, Martin dived deeper into some architecture choices in the collections libraries to illustrate additional crazy Scala language choices and features. But he lost me and, from speaking with some of the other attendees, many others too.

Aaron Bedra: The Future of Web Security isn’t Preventing Attacks

Senior Fellow, Groupon

I quite enjoyed Aaron’s talk. He had a lot of energy, which was great for an afternoon talk on the final day. Aaron made his point right away that you cannot prevent attacks any more than you can prevent someone from mugging you on the street. We should stop treating security as purely preventative. He used the analogy of a casino, and our web applications already ID visitors like casinos. However, unlike casinos, once a visitor is in we stop caring and assume its all ok! Casinos are watching everyone, everywhere, all the time.

So what do we do? According to Aaron, the first thing to do is to aggregate your logs, drop a query engine on top, and generate graphs. Beyond tools, we should analyze what the risks are in having our security breached, and have a well-thought out plan for when breaches occur. Do you have a plan? We should not design security without intent, and security should be part of the design process of our system. Security should be responsive, intelligent, and focused.

Great security should not only tell us what, when, and how, but also who and why. This is what we should strive for. The attackers will always be one step ahead of us. So instead of focusing on prevention, we should focus on getting great at detection and response.

Meet LazyDoc

Wed, 25 Dec 2013 19:44:00 GMT

I’ve been pretty heads down in a ThoughtWorks project for the better part of 2013 and have unfortunately neglected this blog. But 2014 is a new year and I want to get into a semi-regular cadence of posts.

So with that in mind…… another long overdue post! So say hello to LazyDoc, a Ruby gem I have released. LazyDoc provides a DSL for extracting deeply nested values from a JSON document.

Working with APIs

In many projects I work on these days, the application consumes a 3rd party or internal API. In Ruby, it’s tempting to just parse a JSON response from an API into a Hash and then pass that around your application.

def extract_user_from(json_body)
  JSON.parse(json_body)
end

This “works”, but now your application is tightly coupled to the structure of the API response. The fields your application needs are never defined, making it tough to know what you have available to use. Is the “name” field “first_name”, “firstName”, or just “name”?

Ok, easy enough solution for that. Objects to the rescue!

class User
  attr_reader :first_name, :last_name

  def initialize(first_name, last_name)
    @first_name = first_name
    @last_name = last_name
  end
end

...

def extract_user_from(json_body)
  body = JSON.parse(json_body)
  User.new(body['first_name'], body['last_name'])
end

Now we have separated the JSON response from our application domain. Sweet! Ok, done… Right? Not quite. There are a few problems here.

What about everything else in json_body? It is now lost. :(
What do you do when you need to extract more information from that JSON response? This code isn’t very flexible. Either you end up with a constructor that takes in too many arguments, or you end up with too many setters on your User object.
Many times you want to massage your API response data a little bit before adding it to your object. These massaging operations end up going into random helper methods that end up being duplicated all over your codebase.

We can do better.

Embedded Document Pattern

The Embedded Document Pattern to the rescue! Defined by Martin Fowler here. With this pattern, instead of parsing through your documents when they are received and building object graphs, we cache the document and only parse through it when necessary. Because sometimes your document may be large compared to the number of properties your application actually needs, you can save complexity and time.

The LazyDoc gem is a Ruby implementation of the Embedded Document Pattern. You maintain access to the entire JSON response, which makes it easy to add, update, or remove fields from your object as you need. You retain flexibility. Any massaging that needs to be done to the fields becomes easy to manage.

So now, with LazyDoc:

class User
  include LazyDoc::DSL

  access :first_name, :last_name
  access :address, default: 'NONE PROVIDED'

  def initialize(document)
    lazily_parse(document)
  end
end

...

def extract_user_from(json_body)
  User.new(json_body)
end

Lazy?

So why is it called LazyDoc? The lazy feature is one of the coolest parts of this gem. The declarative method access :first_name merely defines a method that is able to fetch the first_name property from the JSON body. Only upon calling the method will the property be extracted and massaged according to any defined operations. Additionally, the returned value will be cached, so any subsequent calls will only return the cached value.

Check the README for more information and examples. Also, the acceptance specs have full example usage.

Feel free to download and use in your next project! Feedback is welcome!

Fun With Rails I18n

Tue, 29 Jan 2013 20:00:00 GMT

Skipping the Basics…

Anyone wanting to develop a truly internationalized application needs to address translating the text of the app into the supported languages. Fortunately for you Rails devs out there, the framework provides an easy mechanism for managing and performing those translations, I18n.translate or shortly I18n.t being the forefront. I am just going to mention the basics here and then move right on to some of the more fun features of Rails translations that every Rails ninja needs to know.

So we all know how to do basic translations:

en:
  hello-world: 'Oh hai, World, you are looking nice'
  views:
    welcome: 'Welcome, %{user_name}'

t('hello-world')  # Oh hai, World, you are looking nice
t('views.welcome', user_name: 'OptimusPrime')  # Welcome, OptimusPrime

Now some things you may not know about.

Pluralization

When you do translations you don’t have to try to hack together interpolated strings using ActiveSupport#pluralize. Pluralizations are baked right in.

en:
  views:
    messages:
      zero: 'You got no messages here fool!'
      one: 'Only 1 message right now.'
      other: 'You have %{count} messages.'

t('views.messages', count: 0)  # You got no messages here fool!
t('views.messages', count: 1)  # Only 1 message right now.
t('views.messages', count: 5)  # You have 5 messages.

HTML Safe

As a general rule, I avoid putting HTML markup in a translation string. HTML markup should be in the view where it belongs. However, despite my best efforts, sometimes it is unavoidable - which is okay in certain situations. Furthermore, sometimes the variables passed into a translation will contain HTML markup. The default strategy in this situation that I have seen is to use raw or html_safe. While this does the job, it adds unnecessary method calls when the HTML safe-ification can be handled directly by I18n using the _html suffix.

en:
  views:
    account-will-be-locked_html: 'Your account is about to be <strong>locked</strong>'
    remaining-characters:
      html: 'Remaining characters: %{amount}'

t('views.account-will-be-locked_html')  # 'Your account is about to be <strong>locked</strong>' (marked HTML safe for the view)
t('views.remaining-characters.html', amount: '<em>50</em>')  # 'Remaining characters: <em>50</em>' (marked HTML safe for the view)

Though you do receive the added benefit of avoiding unnecessary html_safe calls, the real winner here it that by being explicit in the locale file, you tell any other developers looking at it which strings are expected to have HTML. Easy win for visibility and communication.

Watch Out for yes/no!

This is a weird one. I am actually not sure why this happens, and would appreciate if anyone can shed some light on the underlying reason for this. If you try to use the key “yes” or “no,” I18n.t will not be able to find it.

en:
  views:
    yes: 'Sure thing'
    no: 'No way!'

t('views.yes')  # <span class="translation_missing" title="translation missing: en.views.yes">Yes</span>
t('views.no')  # <span class="translation_missing" title="translation missing: en.view.yes">No</span>

Only solution I know is just to not use “yes” or “no” as keys, which is not ideal if you are adding a translation for those exact words.

Literal Naming FTW

It can be quite tempting to use semantic keys in your locale YML files. I see things like “title,” “introduction-1,” or “header-text.” While this will work fine, I prefer to use literal keys that reflect the actual content of the translation (in the default locale) so my views are easier to read for myself and other developers.

- # Hard to read. I have to jump out of my code just to read my own code! Ergggggghhhh
.header
  = t('views.header.user-intro', user: @current_user)

.main-content
  %p
    = t('views.common.site-description-1')
  %p
    = t('views.common.site-description-2')

  %ul
    %li
      = t('views.common.user-perks-1')

- # Easy to read. I understand my own code. Yessssssssss
.header
  = t('views.header.welcome-to-the-site', user: @current_user)

.main-content
  %p
    = t('views.common.we-have-doo-dads')
  %p
    = t('views.common.and-foo-bar-widgets')

  %ul
    %li
      = t('views.common.receive-service-any-time')

That’s all for now! Most of this plus everything else you wanted to know about i18n can be found in the Rails Guides.

Bye! Adios! Adieu! Aloha!

Rails Controller Specs Don't Always Play Nice With Hashie

Wed, 26 Dec 2012 14:58:00 GMT

Hashie and Rspec - The Problem:

Hashie is a neat little Ruby gem that extends Hash and gives object-like access and functionality to hashes. Classes can extend from Hashie and add other functionality as needed. It is especially useful when marshaling JSON or XML data from a service layer into your business models.

While Hashie is very useful, we have to be careful using this gem with ActionController Rspec tests. When creating the assigns hash used in controller tests, Rspec creates a HashWithIndifferentAccess, which is dangerous with objects that act like Hash (such as Hashie objects). Rspec ends up converting user-defined objects that inherit from Hashie into HashWithIndifferentAccess objects, causing us to lose any data that exists outside of the backing hash.

The Setup:

Lets imagine our Rails application talks to an Employee API exposing JSON data. We have already implemented the service layer that makes the API call and returns Employee objects that extend Hashie. In the EmployeeController we make the service call and assign the resulting Employee object.

# employee.rb

class Employee < Hashie::Dash
  property :first_name
  property :last_name

  # foo is not specified using "property" because it does not come from the EmployeeService
  #   JSON response. We are separating what comes from the service and what does not.
  attr_accessor :foo
end

# employee_controller.rb

class EmployeeController < ActionController
  def show
    @employee = EmployeeService.find_by_id(params[:id])
    @employee.foo = 'extra info'
  end
end

The Tests (Where the conflict occurs):

We should be able to write some simple specs to test the controller, specifically, that the result of the service call is stored in the correct variable passed into the view and that we assign whatever extra processing we need to do outside of the service layer into “foo.”

# employee_controller_spec.rb

require 'spec_helper'

describe EmployeeController do
  describe 'show' do
    it 'should assign the employee' do
      EmployeeService.stub(:find_by_id) { Employee.new }

      get :show, id: 'employee_id'

      assigns[:employee].should be_an Employee
      assigns[:employee].foo.should == 'extra info'
    end
  end
end

You would assume this test would pass right? Nope! Fail!

The output of both assertions would be:

expected {"first_name"=>"John", "last_name"=>"Smith"} to be a kind of Employee

NoMethodError: undefined method 'foo' for {"first_name"=>"John", "last_name"=>"Smith"}:ActiveSupport::HashWithIndifferentAccess

As you can see, Rails has converted our Employee object into a HashWithIndifferentAccess! How dare you Rails?!

Why would Rails do this?

Well, don’t be too quick to point the finger, this is actually a combination of Rspec and Rails (ActiveSupport). Rspec is trying to make it easier for you to use the ActionController view_assigns hash by converting it into a HashWithIndifferentAccess. Diving into ActionDispatch::TestProcess we find the method definition for assigns:

# test_process.rb

module ActionDispatch
  module TestProcess
    def assigns(key = nil)
      assigns = @controller.view_assigns.with_indifferent_access
      key.nil? ? assigns : assigns[key]
    end
...

The easy access to the view_assigns hash is great, but looking deeper into the internals of the with_indifferent_access method we find that while creating the HashWithIndifferentAccess, any assigns variable that is a Hash is converted into a HashWithIndifferentAccess! Since Hashie classes return true when asked if they are a Hash, they also get converted, therefore, losing their original object identity. To me ActiveSupport is stepping over the line here. Yes, give us easy access to the assigns hash, but don’t mess with the actual values of that hash unless I tell you to.

Easy workaround:

First, this problem is unique to Rspec tests - production code does not have this same problem. One solution would be to monkey patch ActiveSuport::TestProcess and ActiveSupport::HashWithIndifferentAccess. But, since I try to stay away from monkey patching whenever possible, the quick solution is just to not use assigns when your object under test is a Hash and you care about it not being converted to a HashWithIndifferentAccess for testing purposes.

Instead, use the controller’s view_assigns hash directly to avoid the conversion to HashWithIndifferentAccess:

# employee_controller_spec.rb

require 'spec_helper'

describe EmployeeController do
  describe 'show' do
    it 'should assign the employee' do
      EmployeeService.stub(:find_by_id) { Employee.new }

      get :show, id: 'employee-id'

      # Can not use assigns[:employee] here because Employee inherits from Hashie
      controller.view_assigns['employee'].should be_an Employee
      controller.view_assigns['employee'].foo.should == 'extra info'
    end
  end
end

In the end, its an easy workaround for a slightly annoying “feature” of Rspec.