Today, our speakers, Lauren Knausenberger, Enrique Oti, and Jeff McCoy, have commandeered the term “agile AF”—it stands for agile Air Force, in this instance. Their presentation on DevSecOps in the federal government details exactly what that looks like.
Many people don’t believe that the Department of Defense (DOD) can go agile. But when they see it, they’re inspired. If the DOD can go agile, then anyone can.
First, Oti tells us a story about a project that starts back in Silicon Valley, where federal employees see their friends writing code and making millions of dollars. They wanted to know, “Why does the DOD software suck?” So Oti saw an opportunity and simply asked for resumes. He recruited six airmen and brought them to Pivotal Labs, where they learned to design and then code an app.
So what did they write for their first application? They looked at a problem that had previously been solved by a whiteboard. They looked at how to refuel planes in the air and how to plan and schedule this process. The traditional solution of using the whiteboard, with its flexibility and other benefits, was a formidable challenge to replace. They were tasked to build something better than that whiteboard! In four months they had built a successful application.
McCoy was one of the six airmen pulled in. His team, coming together from different backgrounds and organizations, had kept up their camaraderie and stayed in touch. Though there were conflicts and challenges, they grew stronger working together. Both the Air Force and the Pivotal Labs engineers learned from each other and grew their skills.
This application was a success not just because of the resulting product. The team overcame issues in a place where there were running jokes about how you can’t write apps in the Air Force. They didn’t have IDEs or tools. They had to write their code in Notepad and other simple editors, as everything was blocked for possible security threats. They were eventually able to get around that, and they credit modern DevOps tools for providing one of the biggest wins for Air Force application development.
Along the way, the Air Force realized that they could theoretically be deploying code five times a day. The problem is, each time they deploy, they have to go through an authorization to operate (ATO) process, which takes as little as six months and as much as 18 months. So they worked to adopt commercial best practices to build a continuous ATO process.
Eventually, they got to a point where even people Knausenberger referred to as “supervillains” said that the Air Force was good with their continuous security integration. When the supervillains say you’re good, you know you’re good.
The first org that tried this was the NGA, and the Air Force took advantage of their process. They took a release process down to 22 days, which then showed the opportunity to drop it down even more. They drove against the bureaucracy that they were seeing.
Removing bureaucracy wasn’t just done with pure force. They recruited the right people and created the right professional network to further the DevOps efforts. They realized that bringing everyone together to solve the problem would lead to better results.
Also, what they deployed worked, and that matters. You can’t put together a massive plan for a massive budget: that would have not survived bureaucracy. So they simply built apps and shipped them. It was pretty on the front, fast and dirty on the back, and the customers loved it. They didn’t build for scale at first and focused on getting something to their users. They focused on a minimum viable product in the early days.
Recently, they’ve strayed from MVPs and have become perfectionists at times. But lessons from the old days show them that there’s value in quick and dirty to get something out in front of the users. Do something amazing and do it fast.
Sometimes McCoy hears complaints about the architecture of some of the early systems like JigSaw, but he emphasizes their goal at the time. They were trying to get something out that works. They had a lot of people cycling through the product team. All of this results in less than ideal architecture. But it was still successful. The focus was on delivery.
By delivering something that users loved, fast, they showed that there was a better way: that you can build great systems within a bureaucracy. Sometimes you have to go fast to prove the value, and then people will change. You don’t fight the bureaucracy; you revolutionize the organization by working with others to show them a new way.
Some of the focus has changed over time. For example, at SpaceCAMP and Platform One, some of the work now is hidden from the users. The work focuses on building a platform for other developers to help them move fast and deliver applications.
As you can see, the journey for the Air Force started with a quick and dirty drive that focused on solving problems for customers. They learned to prove their hypotheses and iterate on solutions. And now they’re focusing on enabling other teams to do the same.
This post was written by Sylvia Fronczak. Sylvia is a software developer that has worked in various industries with various software methodologies. She’s currently focused on design practices that the whole team can own, understand, and evolve over time.
Photo by Aaron Barnaby