Ask Me Anything Keynote: CI/CD

Sacha Labourey, CEO of CloudBees, and Dinash Keswani, Group CTO for HSBC answered questions about continuous integration and continuous delivery.

Making Chairs

A better way of talking CI/CD may be to call it continuous everything. The concepts here encompass all parts of software development and release. In Sacha’s mind, releasing software is like making a million chairs. As you make more chairs, the market shifts, and we must shift our process and design to match. With software we need to release often and adjust, keeping up with the changes that occur around us. 

Facebook is famous for saying “go fast and break things.” But to Dinesh, when you go faster, you actually break less. You release less and can pinpoint issues quicker. That’s good, because we want to be doing our best every day.

CI and Regulations

But in a highly regulated ecosystem like banking, how can we move fast? We have to follow the necessary controls. CI actually helps such ecosystems move even faster because of shifting left. 

Instead of engineers having to think daily about the protocols and controls, they can be baked into the CI pipeline, automatically running. And the tools we have today are much better than the tools we had years ago. 

If the CI is done well, CD can happen at a faster pace.

Starting Continuous Everything

Here’s a challenging question. How does a team or an org start to practice CI and CD? Sacha says we should always start small. 

What’s the smallest thing the team can get done? Build on that. You can start with unit tests, or maybe you can start with a simple build script. But don’t try to solve all the problems at once.

You don’t need a revolution to start the journey. Once you can demonstrate small successes, you can build on that and gain momentum. 

To Dinesh, starting really depends on the team and the objectives. For example, if you work on mainframes the customer may have a different expectation than if you work on a mobile app.

You always have a north star, a far-off goal the team moves towards. But you have to measure progress. You can only go as fast as the team lets you. 

We should keep in mind that the goal is not to get better tooling; it’s to impact the customer in positive ways because the product is better and faster. And we should be measuring that. For example, one of Dinesh’s teams have a goal of doubling users per quarter. At the end of the day, the question is this: are you ready to drive the change? If so, take a goal-oriented approach, not a tool-centric one.

Pandemic-Driven Cloud

As people get started with continuous delivery, has the COVID pandemic pushed more teams to shift to the cloud for their CI/CD process? Dinesh seems to think so, but it’s hard to say, as that push was happening regardless. 

Sacha agrees and calls out the top cloud initiatives for 2020. This includes automated policy, which helps teams not only go faster but also to go safer. CloudBees has observed an overall growing appetite for cloud. The data center is no longer the crown jewel of IT, and this is dismantling policies centered around them. 

For example, as people switched to working remotely, one of the largest questions was, “How do I connect to my company’s virtual private network?” These networks become less necessary as things move to the cloud.

Differences of CI versus CD

When we talk about continuous everything, it’s important to understand the difference between continuous integration and continuous deployment. 

According to Dinesh, in CI, you must ensure you have a stable codebase through automated tested and compiling. CI mostly automates your mundane tasks and catches issues early on. For example, if you’re working on something and doing a Git pull five times a day for yourself and testing the integration of your changes, then automate that. Start small and see how those tasks reduce your toil.

Deploying is more about how you get your packages into your user’s hands. The cadences for CI and CD can be separate. For instance, you can run your CI pipeline once a day and only deploy once a week. 

The D in CD can actually represent two things: delivery or deployment. The difference is that delivery is about getting your build ready to go into your environments. But that doesn’t necessarily mean you actually deploy regularly. Deployment is about actually putting that code into your environments. 

Deploying may not be a regular thing; it may be based on when features get finished or timed to certain events. 

One more thing: you don’t just decide to move to continuous deployment one day. That’s because this is a big culture shift for many teams, and it’s not appropriate for all software. But you can always move towards continuous delivery. 

Change Management and Automation

Change management controls are common in many organizations. It’s how they attempt to control the risk of deployments. This can often inhibit the journey to automation. 

Yet automation is the only way you can really scale in an organization. The book Accelerate: Building and Scaling High Performing Technology Organizations calls this out. 

Also, companies that automate have 1/7th the risk of companies that don’t. There’s a belief that releasing often increases risk, but it doesn’t. It actually reduces risk, as there are fewer changes to investigate and recover. 

This can save you much effort—specifically, spending days investigating a multitude of changes. It’s not only more difficult to investigate many changes at once; it’s also that the impact can be much larger. After all, these feature build on one another, and one broken feature can ripple into a snowball of problems when you’re releasing slow. 

When we release small, we can quickly roll back when we detect a negative impact on customers. We can also release features to subsets of customers, starting with 2%, 10%, 50% then 100%. We can watch the software in between and look out for issues. That’s why the ability to quickly roll back is important here. 

Continuous Everything With Management

A key challenge to doing continuous everything is convincing management of its necessity. It’s important to call out the value stream gains they will see from this. 

Managers want to see a reduction in toil. And they want to see costs saved. We can show that with automation. 

In one of Dinesh's teams, they reduced the pain of spinning up infrastructure, like servers. He was able to show the gains the team made by taking this pain away. 

Moving towards continuous everything with management requires champions—people who can continuously push the status quo and help show the victories teams make through automation. Change comes with fear. That’s why we have to have people who can pinpoint that fear and then show them how those fears won’t come to pass. Rather, that which they fear can actually be a huge improvement. 

Conclusion

In continuous everything, we’re moving towards value-added CI and CD. It’s not just about IT anymore but rather about the whole organization. Teams should start small, be goal-oriented, and build on their victories. 

Once you get started, set milestones that keep driving you forward. Make it a community-based effort. Doing so will put continuous everything more in reach than you ever thought possible. 

This session was summarized by Mark Henke. Mark has spent over 10 years architecting systems that talk to other systems, doing DevOps before it was cool, and matching software to its business function. Every developer is a leader of something on their team, and he wants to help them see that.