DevOps-as-a-Service: Overcoming Challenges in Large Organizations

DevOps transformation can require a major shift in organizational culture and “the way things are done.” This can be difficult in any organization, but gets incrementally more difficult as the size of the organization grows. When you get to the size of the U.S. government, implementing DevOps can be seemingly insurmountable. But, it can be done.

For many organizations, especially a government agency, security controls can be a chief focus and one of the “reasons not to implement DevOps,” because security professionals mistakenly believe handing control over to others or automating the process will weaken security. Done well, it makes applications more secure. But the hurdles can be high to implement the necessary changes, especially in a bureaucracy where process and hierarchy trump efficiency.

Janek Claus and Svetlana Yazhuk work for General Dynamics Information Technology, implementing DevOps for clients. Svetlana is a DevOps engineer supporting a platform for continuous deployment of containerized applications for a large U.S. government agency. Together, they share challenges overcome and the platform they are using during their presentation at last year's All Day DevOps conference.

Janek introduced several challenges to improving application security with DevOps. They are applicable to any large organization, with some specifics to the U.S. government.

Talent Shortages

Janek references a U.S. government report on the challenges on hiring and retaining enough security specialists, noting that 74% of agencies are “At Risk” or higher, according to the report. Of course, this is true across many other industries and organizations. Governments also see recruiting and retention challenges because employees want to work in modern environments. Training employees is also difficult because - as we all know - it interrupts the work everyone wants to do.

Some solutions Janek covers include:

• Bake automated security into the pipeline
• Give developers and testers security tools
• Use continuous security
• Gamify training, such as code bashing

Functional Silos

If you spent at least 10 minutes learning about DevOps, you have heard about the need to breakdown silos in the organization. Silos separate teams and hand-offs complicate the process. Two solutions include:

• Modeling your organization with the principles of Conway’s Law - structure an organization that resembles your goals
• Creating a platform team that offers DevOps-as-a-Service

Hesitation to Change

People naturally hate change. It is a law of nature. You can’t change that, but you can show people the value of the change so the desire to change overcomes natural resistance. Changing culture can be especially challenging, but it has to be done to properly implement a DevOps culture. As the famed business strategist Peter Drucker said, “Culture eats strategy for breakfast.”

What can you do to change the culture:

• Ensure sustained executive management support and a long-term vision
• Provide training, coaching sessions, etc. on tools, concepts, and concrete implementation steps, but make sure to provide them in a way that offers the least amount of interruption to the normal workflow

Authority-to-Operate

The Authority-to-Operate (ATO) is a specific term for the U.S. government, but the concept can be found across organizations. In essence, security needs to approve an application, or even a subset of a larger application, to operate on the system.

For instance, you might need to get an ATO for a library you want to use it in your larger application. This can be a time and resource consuming process that can take months or even longer. It also hampers the availability of the best tools for the solution. If you can leverage cloud solutions or, even better, DevOps-as-a-Service, the ATO for these services can cover many of the items you need. This eliminates the need for separate ATOs for each application.

Secure DevOps Tools Integration

Don’t forget that your DevOps tool chain needs to be secured too. With dozens of tools being used, each with their own privileges and credentials, the security risk grows. Employing secrets management and solutions that offer integrated credentials management across the DevOps pipeline help ease the burden and tighten your security posture.

Secure the Software Supply Chain

Janek notes these days that 80-90% of an application’s code may not be your own code. Commercial and open source components help make application development efficient, but you need to take steps to ensure that code’s vulnerabilities are managed, continuously. Ensure you have tools that scan your libraries before they are used and continuously monitor applications during development, testing, deployment, and in production. Also, leverage existing open-source intelligence and use tools that inject human intelligence into scans.

DevOps-as-a-Service Platform

Janek and Svetlana are implementing DevOp-as-a-Service to solve the aforementioned  challenges. It involves a team and a platform with the goal to continuously increase the number and scale of self-service capabilities.

Svetlana dove into some specifics of the platform, Monarch, during the presentation. Monarch is a platform that GDIT created for their government client. It is a self-service platform that development teams can use to build Docker containers and deploy them into AWS Elastic Container Service.

Currently Monarch has:

• Early vulnerability detection
• Quick remediation response
• Frequent updates in production are encouraged
• Multiple parties involved in security auditing

Next steps including providing code quality monitoring via Sonarqube.

You can listen to Janek’s and Svetlana’s entire presentation, including a more detailed explanation of Monarch here.

Register now for the next All DayDevOps, November 6, 2019.

photo: RawPixels