Fannie Mae operates in the already heavily regulated financial services/mortgage markets. Add to that it is a government sponsored entity, which means it is laden with many of the policies and regulations of federal government agencies. Plus, they have 6,000 employees and annual revenue of $22 billion. So, they could be a poster child for, “You can’t implement DevOps there.”
Barry Snyder, DevOps Manager, Developer Platforms at Fannie Mae, begs to differ, which is what he did in his 2016 All Day DevOps talk, How Fannie Mae Applied DevOps to Deliver Quality at Speed. He looks at:
Fannie Mae started implementing DevOps in 2014, and Barry was in the middle of it from the start. He gave his talk at the end of 2016, and, as he said, they still aren’t fully adopted (are you ever), but he gave a status and a comparison between where they were and where they are now.
|
2012 |
2016 |
|
10 projects using modified-Agile. Essentially waterfall with terminology/concepts from Agile |
155 using Scrum/SAFe and growing |
|
9 to 18 months to release to production |
Projects releasing every month |
|
9 governance gates/inspect 100% before release to production |
1 working group, 1 CAB, inspect 10% after release to production |
|
2-4 months to provision dev and test servers |
Minutes to provision dev and test servers |
|
300-400 builds per month; 900-1200 deployments per month |
3-4K builds per month; 12-15K deployments per month |
|
Testing took 50% the time it took to construct the solution |
80% savings in testing time and executed within the sprint |
|
No means to measure and evaluate the productivity from release to release |
Productivity increasing by an average of 30%-40% with costs going down by 30% |
According Barry, these results are being sustained - they are not just a blip. To get there, they had to get an appetite for risk. Overcoming that hurdle required them to realize when you do 100% risk mitigation, you are going to spend more time avoiding risk than it would take to mitigate something.
They are also executing Application Lifecycle Management (ALM), a step up from CI/CD. They look at all of the tools and processes along the lifecycyle, peer reviews, automated building, testing, etc. They are implementing automated processes with a variety of tools to achieve this.
As an example, at Fannie Mae, it previously took 5-6 weeks to request a library, now it is immediate because it is automated through Sonatype’s Nexus.
As we hear from all practitioners, it is a journey, not a sprint (pun intended). Fannie Mae started at the end of 2014, and they are still in progress. They went from 9-18 months per deployment to a month on a consistent basis. They are close to always deploying during sprints, and are working towards having an idea, writing code, and releasing it into production.
Barry presented lessons learned:
Leverage what they had while shifting left:You can watch Barry’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on October 24th.