Executing Continuous Delivery

Problem

"While leading a team at LivingSocial, I set out to achieve the goal of safe code shipping as a habit but without the complicated and time-costly infrastructure. We were successful by incorporating good software engineering and deployment practices–all of which were generally good for us and didn't require as much dedicated tooling or time. Later we discovered others outside the company were starting to do the same under the label 'continuous delivery.' We have been even more successful with continuous delivery at LearnZillion, where I am today. Unfortunately, the cost of continuous deployment infrastructure can discourage engineering teams from investing time in their development and deployment process because they don't realize the lower-cost alternative, continuous delivery, is also a viable option."

Actions taken

• 0. Assumptions. I am going to assume the year is 2015, or even 2010 or 2006, and that you have a deployment script or tool like Capistrano to automate the basic deployment steps for your application. As well, I'm going to assume your team or organization wants to do continuous delivery. If neither of these are in place, start with them.
• Individual responsibilities. Although we work as a team, individuals are responsible for carrying work forward to completion and doing that well. Staff are responsible for taking features from initial definition to product shipment. Along the way, they collaborate with and incorporate input from the broader team and company. With these responsibilities come expectations.
• Smallest overlap of responsibilities. We often pair a product designer (design, UX, HTML/CSS) with a full-stack engineer (SQL, Rails, Ruby, JavaScript, HTML/CSS) to work on a feature. However, we avoid assigning multiple engineers the same feature. We try to keep engineers working on "orthogonal capabilities." (See "The Three Musketeers" and The Mythical Man Month for the rationale behind this approach.)
• The master branch is sacred. We deploy to production from our master branch. Features are developed, reviewed, and QA-ed in separate branches. If you have test failures, it's most likely your code. Feature branches are only merged into master immediately before deployment. It is the responsibility of the feature owner to make sure the branch is reasonably current with the master before it is merged itself. There are loads of articles on "simple git workflow," which you can find online, like this one. git and GitHub make this paradigm easy to follow.
• Follow "The Twelve-Factor App" methodology. I will let the methodology speak for itself. See part X in particular. The biggest continuous delivery benefit is no surprise during deployment. At LivingSocial, my team ensured the application development environment behaved like production, except where Rails intentionally separates the two. Truth be told, we didn't have a reliable staging environment at our disposal, so we went straight from development to production. At LearnZillion, we take this further by using similar SaltStack configurations for production, staging, and a Vagrant-powered development environment. In development, the Ruby process and gems for the app are still installed on the host operating system but everything else runs inside VirtualBox. It has the side benefit of speeding-up the on-boarding process for new engineers.
• A test suite. At both LivingSocial and LearnZillion, we used Ruby on Rails, which strongly encourages the use of a unit testing framework. Engineers make certain to have a passing test suite before merging a branch into master, must have a passing test suite on master post-merge, and a failure on the master branch takes top priority–second only to a live site outage. At LearnZillion we took this farther by integrating CircleCI with GitHub to minimize the execution burden on engineers.
• An automated QA test suite. At LearnZillion, we have a QA team. They naturally have the potential to be a bottleneck for getting features out. Since quality is their main objective, you want them to be gatekeepers. What you don't want is for their review and gatekeeping processes to be cumbersome or inefficient. The most powerful lever you can maneuver within your QA team for continuous delivery is to automate their testing. Our team has an extensive QA test suite, which QA engineers can run against any branch, at any time, on a staging server. Automated tests are usually written soon after deployment to production but sometimes are completed before then. Manual QA of emerging features still takes place, of course.
• Look at your dashboards. It doesn't take much effort to have a short list of links to Google Analytics, Mixpanel, or your error reporting services like Bugsnag or Honeybadger. An engineer can inspect them after deploy to see if something broke. Engineers and product designers should be doing this anyway to see how users are responding to changes or new features. BONUS: Manual QA in a different time zone. When an engineer's code has passed peer review and the automated QA test suite, it is sent along to QA for manual inspection. Test results are back by the next business morning because some of our QA team members are located in India. They test our work while we sleep. BONUS: Continuous QA. At LearnZillion, we've integrated a GitHub pull request webhook that deploys a branch to a staging server and runs the QA test suite against it. This means that a branch has been regression tested before it gets to the QA team and usually before it gets to peer review. If you want to read more about our automated QA process, see Kevin Bell's article about us over at CircleCI.

Lessons learned

• Unfortunately, I have found that in many organizations, a lack of trust is the default. A tech lead or manager is responsible for scrutinizing code from all team members, merging, deploying, and ensuring the application won't break. This may make sense for new team members until they understand and are comfortable with the team conventions and have demonstrated that they are capable engineers. Otherwise, it should not be the norm.
• Developers can depend on master as a reliable foundation to fork, merge, and rebase from.
• Believe it or not, because of our practices, going directly to production from development still worked quite well.
• With the good engineering and deployment practices of continuous delivery, you can achieve the same benefit of continuous deployment: safe, consistent delivery of product as a habit. You don't have to build-out a dedicated infrastructure, and you can build a better engineering team and environment in the process.

Source: Ian Lotinsky's Blog