Adapting Scrum to Continuous Deployment

Problem

Several years ago, I joined a development organization that operated on a very inefficient one-month sprint cycle and manual deployment process. This quickly made it the top of my list of things to improve. However, the existing leadership team was very risk-averse, felt the existing processes were good enough and did not want change.

Actions taken

Gathering data to support the changes

Undeterred, I observed the cycle for several months and documented the many pain points: urgent mid-sprint deploys for bug fixes, wasted developer downtime time at end of or in-between sprints, giant monolithic deploys that were hard to debug and rollback, error-prone four-day long manual QA regression, the pressure to hit sprint deadlines leading to painful last-minute shortcuts, a two hour, 25 people, cross-department pre-deploy “go/no-go” meeting and a manual, late-night deploy that took the product out of service. I took this laundry list of issues to the leadership team and got a conceptual agreement that the process needed to change, but no resources or official support.

Guerilla effort to demonstrate early results

Again, I was not deterred. I set a goal of enabling fully automated deploys at least once per day and created a roadmap to get there. I developed the first several milestones of the automated deploy pipeline myself on nights and weekends. At the same time, I worked with members of the QA team during their downtime at the start of each sprint to develop an automated QA regression test. I also developed and rolled out a distributed risk management process to eliminate the need for the 25 people “go/no-go” meeting.

The demonstrated value of the partially complete deploy pipeline and automated regression tests allowed me to elevate the automation project to an officially sponsored initiative. With the ability to commit dev teams to work on the project I was able to complete the deploy pipeline, extend the regression test suite and make the significant application changes required to allow a zero-downtime deploy.

Small experimentation and iteration

After achieving the goal of daily fully automated deployments, the heavy sprint cycle development process was the next bottleneck slowing us down. Over the course of a year, I experimented with process tweaks on a single team, rolling out each successful change to all the other teams. Eventually, the vast majority of the existing process was stripped away and replaced with a lighter weight process. This process, which we still use today, allows teams to deliver a predictable flow of work while retaining the ability to quickly react to production incidents and changes in project scope.

Lessons learned

• Use data to get buy-in to change entrenched practices. While the solutions seemed obvious to me, data and demonstrated improvements were what really convinced leadership to back the project.
• Don’t settle when you believe you are right. Despite the lack of management support, I continued pushing the project because I believed in it. If I had given up, the organization would likely still be doing things the same, inefficient way.
• Start Small. When changing processes/tooling, test it out with the smallest unit you can (probably a team.) Iterate at that level and tune it before fanning out to a larger group.