One Team, 5,000 Jobs: Life in the DevOps Jungle

Damien has 5,000 jobs. While you might gasp at that workload, Damien is not stressing out. All 5,000 jobs are automated within his team’s Jenkins pipelines. How does he do it? Damien follows four key principles to keep his cool in the job jungle: self-service, security, simplicity, and extensibility. But you might be surprised that one of his most important survival techniques is treating his pipeline as “not code.”

At the recent All Day DevOps Conference, Damien Coraboeuf (@DamienCoraboeuf, nemerosa.com) discussed how his organization got out of the job jungle. Even with 5,000 jobs in Jenkins, Damien has a tiny team of six running it while supporting a large band of developers around the world.

During his session, Damien laid out how a pipeline, the backbone of a Jenkins job, is constructed. Spoiler alert - it is pretty simple, hence the magic of Jenkins. However, it gets complicated quickly, so Damien shared four principles his team strive towards:

• Self-service. They don’t want the Jenkins team to be a bottleneck, and they don’t want to have to deploy an army just to run Jenkins.
• Security. With teams of developers, many of whom they have never met, security and control is critical.
• Simplicity. Developers can focus on developing rather than Jenkins.
• Extensibility. As they grow and capabilities are dreamed up or implemented, their system needs to be able to adapt.

Core to their implementation and their goals of self-service, simplicity, and extensibility is the Job DSL Plugin. While Jenkins has a robust user experience, as jobs are added, it can get cumbersome and clumsy. The Job DSL Plugin brings order, automation, and consistency to the jobs. Damien walks through the basics of the plugin and its benefits, noting that it treats the pipeline as code.

Treating the pipeline as code allows the team to reuse code through versioned libraries and easily manage thousands of jobs. Requiring developers to manage more work as code does offer benefits, but security risks also come along with giving that many people access to the code. So, Damien’s team actually treats the pipeline as “not code,” by describing the pipeline using properties files. It also allows them to use the pipeline as metadata, so queries and reports are almost enjoyable.

Damien’s team also utilizes the Seed plugin, which they maintain, to, “help automate the generation and management of pipelines,” tighten security by keeping projects locked to teams, and increase automation through hooks.

The Seed plugin shares some functionality with the popular Pipeline plugin, which is maintained by a large community. However, the Pipeline plugin does not allow for treating the pipeline as “not code.” Damien’s vision is that one day Seed will be an extension of Pipeline.

Damien’s team has done a tremendous job of running Jenkins across the enterprise. In fact, he just wrote me with an update that since November, they have scaled to 7,000 jobs while still following the same principles.  If you want to learn more about how they do it, be sure to watch his full All Day DevOps conference session (only 30 minutes). The other 56 presentations from the All Day DevOps Conference are available online, free-of-charge here.

This blog series is reviewing sessions from the All Day DevOps conference from November which hosted over 13,500 registered attendees. Last week I discussed, “DevOps: Making the Boring Things Stay Boring”. Next week, look for “Docker Inside/Out” -- a discussion by Daniël van Gils.