Damming a 97 Year Old Waterfall: Transforming to DevOps at State Farm

Jeremy Castle and Kevin O’Dell, DevOps leaders at State Farm Insurance, help teams understand a new way of working that creates high performing teams through a DevOps mindset. Today, they’ll take us on a journey through their DevOps transformation at State Farm.

State Farm is the 36th-ranked Fortune 500 company and has been around since 1922. They have 83.4 million policies and accounts. Their enterprise technology department alone is composed of 6,000 employees and over 2,000 web applications. There are more than 1,200 product teams to support their systems.

Workspace to Production in 24 Hours

On Jeremy’s first day on the job, his boss challenged him to get code from a developer workstation to production in less than 24 hours. After many years of never seeing projects make it to production, this seems like an overwhelming initiative at first. But it was a simple, tangible challenge that gave him and his teammates a North Star to drive towards.

To see how to change, he had to figure out how State Farm works. So he created a value stream map of the entire process State Farm takes to bring code from inception to production. In this, there were close to 155 processes with almost an equal number of handoffs to other teams.

The value stream map from “Damming the 97 Year Old Waterfall: Transforming the Enterprise to DevOps at State Farm”

It would take over 1,500 hours to get code all the way through this value stream. But at least now they could physically get their heads around how software gets delivered. Some bottlenecks were obvious to see. Others lurked deeper. This value stream map laid the foundation for future change.

The 2015 DevOps journey at State Farm from “Damming the 97 Year Old Waterfall: Transforming the Enterprise to DevOps at State Farm”

There are two mountains they had to climb over to get their 24-hour-to-production goal. First, there was the lack of automation in many of their processes. But they also had to deal with their change processes, at both the department and project levels. Often the only reason these processes existed was because “this is the way we‘ve always done it.”

So they reset the process from the ground up. They automated much of the change processes, lowering production time from days to hours.

The change process at State Farm from “Damming the 97 Year Old Waterfall: Transforming the Enterprise to DevOps at State Farm”

How State Farm Improved

But how exactly did State Farm enable this improvement to happen? Well, when you have all this automation and security, you can’t forget about the people. State Farm had to get their people to adopt the DevOps way of working.

One way they created a DevOps culture was to adopt the “dojo model” that Target had. It’s not a classroom but rather an open place for learning based on the knowledge people currently had. This type of change is the type that will get executives excited, and that’s what happened at State Farm—the developers there found support from executives.

With the automation and the people’s buy-in in place, they next needed to look hard at their team structures. They decided to go from projects to products. Products never end and need to have full support and ownership, and they designed that into their teams.

The 2019 DevOps journey at State Farm from “Damming the 97 Year Old Waterfall: Transforming the Enterprise to DevOps at State Farm”

Challenges

Climbing the DevOps mountain was not without its avalanches and steep hills. They still had dates and people who were used to waterfall projects.

Here are the top five things that they encountered.

1. When everybody owns something, nobody owns it. They were playing feature “plinko,” which meant people were working on things they didn’t understand well. So more ownership was pushed to each team in building their own pipelines, logging, and tooling. The biggest challenge were the product owners, who had to learn more deeply the products their team worked on.
   
   
2. It might be embarrassing. Your first releases will be pretty tough. You have to take the time upfront to build in the automation and resiliency needed to produce the new functionality, and that may mean lower initial features released.
   
   
3. It’s tempting to have a separate DevOps team. State Farm had to push hard to help enable teams to adopt DevOps ideas without siloing into a team solely focused on Ops or on building the automation pipelines. Leaders had to enable and coach, but the product teams execute on the ideas.
   
   
4. You have to flatten the layers. Teams need to be cross-functional so that communication is clear and effective. For example, architects need to be embedded in product teams, not as a separate layer above them.
   
   
5. Learn to unlearn. There’s so much old waterfall thinking that people had to overcome. And people had to be intentional about changing to a different way of thinking. For example, instead of obsessing over code duplication, they needed to focus on decoupling that code.

Automation vs. new functionality from “Damming the 97 Year Old Waterfall: Transforming the Enterprise to DevOps at State Farm”

Summary

Despite the challenges State Farm faced, they had the right leadership support and passion to continue to push forth to a better way of deploying software. Customers, business stakeholders, and software professionals are all happier as a result.

This post was written by Mark Henke. Mark has spent over 10 years architecting systems that talk to other systems, doing DevOps before it was cool, and matching software to its business function. Every developer is a leader of something on their team, and he wants to help them see that.

Photo by Barb Canale