Creating an AppSec Pipeline in a Week

What is the key to creating a AppSec pipeline in a week? Happy developers and coffee.

Well, let’s be honest - that is the key to every software development project.

Let me take you on a little adventure with Jeroen Willemsen (@commjoenie), Principal Security Architect at Xebia, based on what he presented during the 2017 AllDayDevOps conference. His talk, entitled, Creating An Appsec Pipeline With Containers In A Week: How We Failed and Succeeded, walked through his real-life experience to help the rest of us learn what to do and what to avoid.

The Challenge: Automate security in one week. Jeroen’s client had mainframe systems for data for a physical warehouse infrastructure combined with containers on AWS and Angular, iOS, and Android native apps. Their existing workflow is diagrammed below. Joroen emphasized the importance of implementing a process like this before security automation. You need the basic infrastructure or you will fail.

The client had a wish list of new entries into the security pipeline:

• OWASP dependency to check third-party vulnerabilities
• License checkers to ensure licensing requirements don’t expose you
• Clair vulnerability checker for Docker containers
• Dynamic analysis with BURP suite scanner and OWASP Zed Attack Proxy
• Static analysis (they decided not to do it because it was too challenging for the week)

The Solution. First, they extended the build step by adding dependency and license checkers on top of quality tooling and got feedback FAST! Second, they automated ZAP and BURP. The goal? Make sure everything works in the time it takes to enjoy a cup of coffee.

Next, they had to integrate the findings from different tools. They used ThreadFix to filter out false positives and export vulnerabilities found to JIRA tickets.

Then they ran Clair on the created containers, but they could not run Clair regularly on the registry, integrate with Threadfix, and add whitelists. Jeroen noted that this can be done now with the Clair-scanner from ArminC.

Finally, they embedded their tools in containers, gaining some key advantages:

• Less additional platform complexities
• Can run anywhere (locally/deployed)
• Easy to scale
• Still need to manage the data!
• More assets might contain vulnerabilities

However, it was not perfect - they still had to harden their assets.

Did the solution work? Yes, but it was a bumpy road.

Bumps on the Road

Bump 1: False positives. Once you start automating, you will get a lot of false positives. You have to suppress to them, but using settings and plugins or a database with a framework doesn’t scale well. The third option is to have an API. This is what they did. At the time, they used ThreadFix, and now DEFECTdojo is also an option. No matter what option you choose, the lesson is the more you automate, the more you call the tools, resulting in more false positives. So, start with something.

Bump 2: Legacy APIs. Their automated tests would crash the mainframe, so they stubbed it with the help of teams. They still had to test legacy APIs separately.

Bump 3: Not frustrate developers. They needed to like us to help us further. Jeroen’s advice:

• Give feedback fast
• Automate all the things
• Be part of the team. Whenever you`find something, go to the developer before you put tickets on Jira, show them, and suggest how they can fix it. Be proactive
• Filter and suppress false positives ASAP. If they happen too often, developers will just find ways to work around them
• Use known tooling

Bump 4: Integrating Burpproxy. They did not complete the integration with Burp because of custom builds for containers and, at the time of testing, additional extensions were necessary to have a proper REST API.

Bump 5: False negatives. Security automation does not mean no manual pentesting. Both static and dynamic analysis tools miss context.

Bump 6: Platform team availability.

Later on, they learned they needed multiple pipelines.

To hear more about the lessons learned and advice from Jeroen and his team, you can watch his presentation here. If you missed any of the other 30-minute long presentations from AllDayDevOps, they are easy to find and available free-of-charge here.