The Road to Continuous Deployment

We have all heard the excuses: management won’t go for it; the code is a jumbled mess; it's too large; too many regulatory hurdles. The trek to Continuous Integration/Continuous Deployment has stumbled for many enterprises, but many more each day have made it.

Michiel Rook spoke at the 2016 All Day DevOps conference about his road to Continuous Deployment. He used an example he worked on for De Persgroep Employment Solutions, which operates a number of job portals. This specific project was called the San Diego Project, but it was known internally as the Big Ball of Mud because the code base was such a mess. The image below diagrams the legacy system - the beginning of the road.  

The project was burdened with: infrequent, manual releases, fragile tests, frequent outages and issues, a frustrated team of about 16 people, and low-confidence modifying existing code.  Sound familiar?  Yep, you were not alone.

The team knew something needed to be done, so they set some goals, including:

• Reduce issues
• Reduce cycle time
• Increase productivity
• Increase motivation

Their approach was to take the monolith, build a proxy and add a service, and then keep adding services until the monolith can be thrown away.

Of course, this is a simple explanation; much more went into the trek.

So, in this article, we are going to dig deeper, as Michiel also does in his full talk. 

They started with a foundation of principles to guide their journey:

• Apply the strangler pattern
• Use API first methodology
• Set one service per domain object (job, jobseeker, etc.)
• Migrate individual pages
• Establish services behind load balancers
• Access legacy databases
• Implement continuous deployment
• Utilize Docker containers
• Develop frontends as services

This all culminates to continuously deliver value, something Michiel calls "Continuous Everything."

It starts with Continuous Integration: developing and building/testing, resulting in an artifact each time. Then, Continuous Delivery: building/testing-->acceptance-->production; going into production is a manual process, but the code is always deployable. Finally, you reach Continuous Deployment when the whole process is automated.

Continuous Deployment is advantageous because it offers the following:

• Small steps
• Early feedback
• Reduce cycle time
• Reduce risk
• Room to experiment

Now that Michiel’s project is behind him, he offered the following key aspects of any road trip to continuous deployment:

Only Commit to Master. No branches. You don’t want to delay integration and abuse version control for functional separation. Plus, everything on a branch increases the risk of conflicts and delays integration.

Every Commit Goes to Production.

Use pair programming for code review. This will require discipline, but all development needs to be paired. Mix and match experienced developers.

Quality Gates. Ensure a substantial amount of tests and code coverage.

Feature Toggles and A/B Tests. Determine which version people can/can't see and facilitate A/B testing. But, be sure to keep the number in check.

Dashboards. Display is essential for deployments. Measure everything: KPIs, build times, page load times, number of visitors, results of A/B tests, etc.

DevOps. Mentality is a culture; no more walls between dev and ops. Ownership lies within the team for everything, but this doesn’t mean everyone knows everything.

Automate Repeatable Things. If you need to do something twice, you have done it too many times.

Continuous Testing. Use unit tests and smoke tests to see if a service is live, and always monitor. Exploratory testing is important because you continue to test the most critical paths.

Pipeline as Code. Automate the pipeline.

In the end, the deployment looks like this.

Feedback - DevOps is built on the importance of feedback. One example Michiel had on this project was a large, red flashing light that signaled a build failure. Whenever it went off, that became the number one thing people started working on.

Michiel’s project spanned over one year. In the end, they reduced the total build time per service to less than 10 minutes, significantly improved page load times, increased confidence and velocity, and more. The universal truths of the importance of team acceptance and that change is hard were seen. They also learned, among other things, that the alignment with business priorities is key, ensuring you have the necessary experience on staff is essential, and limiting feature toggles is crucial.

Overall, Michiel and his team made it to Continuous Deployment. At the end of his talk, Michiel did report that, to his dismay, the legacy system they are seeking to replace is also still in service. The road is long, but worth the trip.

If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on October 24th.