How To Fully Automate CI/CD - Even Secrets

Imagine a world where your Continuous Integration/Continuous Deployment environment is 100% automated, including the passing of credentials.

Capital One, a leading U.S. bank, has achieved this, and Andrey Utis, Director, Software Engineering, at Capital One, outlined how they achieved this in this talk, Application Secret Management with KMS, at the All Day DevOps conference.

Andrey started by stating what we all know - security needs to be at the forefront. Additionally, a message we heard over and over at the All Day DevOps conference is to automate where you can automate. That combination can be daunting and seemingly impossible. How can we automate security when someone needs to enter credentials and we can’t store credentials where everyone can get to them? After all, credentials give you access to databases that often contain personally identifiable information and other protected information. A breach could devastate companies and people, whether intentional or accidental.

Andrey’s team uses Amazon Web Services (AWS), so they give instances IAM roles that allow them access to other AWS resources. What is the key (pun intended)? On an AWS EC2 instance, there is a magic IP address to which you can make an HTTP call and it will return temporary AWS keys. Those keys then make the API calls to different database services.

Here is the solution. AWS KMS is encryption as a service. KMS Context allows you to add “salt” to the encryption. You can only decrypt with the same “salt.” KMS Key Policies restricts which IAM role can decrypt with the key/salt. Master KMS keys can be used only to decrypt their own keys. Below is a code sample of a KMS policy.

This allows multiple applications to be on one AWS account while limiting access of developers to the applications they are authorized for.

Here is the actual protocol: 

1. Create a KMS master key (can be shared by multiple apps)
2. Create an IAM role for your server
3. Add KMS key policy that allows decrypt to your IAM role for a specific context
4. Encrypt your secret with the key and context and store the value in GitHub
5. At deployment or runtime, invoke KMS API to decrypt

There is a broader issue with IAM roles because credentials are generated by calling the “magic” metadata IP address 169.254.169.254. Developers in production, even with “read only” access to the instance, could call the KMS API to decrypt the secret. Developers should not be able to generate production IAM credentials at all, so you block the magic IP address with this code:  

To automate it and make it reusable, they created a Chef cookbook, which they call a “briefcase,” to abstract decryption of secrets. They also have the iptables cookbook to block the metadata IP address for all except a whitelist of user groups, such as root and any application specific group that makes AWS API calls (see above code).

Andrey mentioned that Vaultmay have a viable solution soon. They use signed EC2 Identity Document to verify the caller. The current downsides are that it only supports authorization by AMI ID, but should support more soon, and secrets are not source controlled/versioned. As Andrey noted, that is not ideal for “configuration secrets” such as database passwords.

In a follow-up question, Andrey was asked, “What were the lessons that were most critical to learn?” He answered, “This is a new field. Most companies either don’t fully automate or don’t fully secure the entire pipeline. So, there is little information out there.”

Well, there is some more information in Andrey’s complete talk, which you can watch online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.  

Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on October 24th.