Introduction

The following list of best practices was created when I was responsible for designing a CI/CD environment for an engineering community of 15,000 engineers.

Eventually, we ended up with 30 fully automated and managed Jenkins clusters.

These clusters followed a hybrid VM/container auto-scaling architecture that provided us with a minimum guaranteed capacity but was elastic and could expand at a moment’s notice to cover any spikes.

This architecture also catered to legacy projects/systems that needed specific versions of operating systems, such as Windows Server 2008 or Mac build targets.

Release Workflow Management Using Feature Branches, Leveraging Git and Multibranch Development

Build your artifact only once and promote it through all test phases.
Use short-lived feature branches and code reviews.
Restrict access to your master branch to only Jenkins.
Use automated Git merge testing as part of your development workflow.
This allows the master branch to always be in pristine condition.
Release from the master branch only. An alternative is to release from special release branches created from master at specific points in time. It’s up to the team to choose what is best for them.

Jenkins Environment Best Practices 1

Always secure Jenkins.
Use an LTS Jenkins version.
Don’t build on the Jenkins master.
Back up Jenkins Home regularly.
Limit project names to a sane (e.g., alphanumeric) character set.
Use artifact repos to manage dependencies.
Build against clean build environments. Remove older artifacts regularly.
Integrate tightly with your issue tracking system, like JIRA, to reduce the need for maintaining a change log.
Integrate tightly with a repository browsing tool like Crucible/FishEye if you are using Subversion as a source code management tool.
Always configure your job to generate trend reports and automated testing when running a Java build.

Jenkins Environment Best Practices 2

Set up Jenkins on the partition that has the most free disk space.
Archive unused jobs before removing them.
Set up a different job/project for each maintenance or development branch you create.
Prevent resource collisions in jobs that are running in parallel.
Avoid scheduling all jobs to start at the same time.
Set up email notifications mapping to all developers in the project, so that everyone on the team has their pulse on the project’s current status.
Take steps to ensure failures are reported as soon as possible.
Write jobs for your maintenance tasks, such as cleanup operations, to avoid full disk problems.
Tag, label, or baseline the codebase after a successful build.

How a Build Job Should Look

We need more than one build job for each project.
Each build job can be used for a different purpose.
Our goal is to make the development cycle as fast as possible.
Using more than one build job allows us to fine-tune the various build phases.
The lightweight phases should be executed on commit (i.e., unit testing).
The expensive operations can be scheduled to be executed daily.
This approach allows us to have special-purpose artifacts that can be analyzed by various quality tools; these are not our releasable artifacts.
Build jobs can be linked to each other, creating build pipelines.
A single build job can have multiple stages.
Using multiple stages allows us to identify bottlenecks in our build job.
Build stages are directly supported by the Jenkins Pipeline plugin.
Using Jenkins Pipeline parallelization allows us to describe the full pipeline in one build job that will fan out for the parallel phases and fan in at the end, collecting all results and publishing them to the appropriate services. This approach combines the best of both worlds—pipeline speed with automation—but assumes that the team is well versed in using Jenkins Pipeline DSL.

A Typical Set of Build Jobs for a Project and Their Expected Stages

A main CI build job. Executed automatically on commit. Stages: Workspace cleanup, code checkout, build, unit test, push snapshot artifact to artifact repo, notify if failure.
A Sonar job. Executed automatically at least once per day. Stages: Workspace cleanup, code checkout, build instrumented artifact for Sonar, analyze, push analysis to SonarQube server, break build if poor quality found, notify for broken job.
A WhiteSource job. Executed automatically at least once per day. Stages: Workspace cleanup, code checkout, build artifact, analyze, push analysis to WhiteSource server, break build if poor quality found, notify for broken job.
A WhiteHat job. Executed automatically at least once per day. Stages: Workspace cleanup, code checkout, build artifact, analyze, push analysis to WhiteHat server, break build if poor quality found, notify for broken job.
An integration testing job. Executed automatically at least once per day. Stages: Workspace cleanup, checkout deployment code, execute deployment code against integration testing systems, set up integration system, pull all artifacts from artifact repo, start services, start testing, record results to Jira, notify on test failure.
A release job. Executed manually or automatically according to release policy. Stages: Workspace cleanup, code checkout, build artifact, push released artifact to artifact repo.
In addition to the above build jobs, we may have multiple other build jobs that test different aspects of the service. After release, all other testing needs to be done against the released artifact. The only jobs that may use source are the primary building jobs that analyze the code and do not produce releasable artifacts.

Build Job Automation Best Practices 1

Automate as much as possible.
This can be done by either using Jenkins Pipeline or Jenkins DSL plugins.
Develop your pipeline as part of your project.
The Jenkins Pipeline should be used to drive the build. Call all the external build tools needed to build the project. Ideally, we should use wrapper scripts (e.g., gradlew, mvnw) to increase portability of the build environment.
Version control your pipeline.
Group common build functionality into external libraries that can be used across projects.
This improves consistency across projects and promotes standardization/automation.
Consider using the NEP Jenkins Pipeline library for this.
Do: All work within a pipeline stage.
Do: All material work within a node.

Build Job Automation Best Practices 2

Do: Work you can within a parallel step.
Do: Acquire nodes within parallel steps.
Don’t: Use input within a node block.
Do: Wrap your inputs in a timeout.
Don’t: Set environment variables with env global variable.
Don’t: Embed scripts in the pipeline. Keep them external and use the pipeline to call them.
Do: Prefer stashing files to archiving.

Test Automation

Automate all tests.
Use test automation services such as Zephyr.
Integrate with Jenkins and Jira.
Provide both current state visibility and historic state of test executions.
Test during development either on commit or daily.
Modify the definition of done for the dev teams to include (smoke) testing against a production-like environment.
Test not only artifacts but also deployment and migration scripts as well as environment definitions.

Managing Internal and External Dependencies

Do not depend on source code.
Depend on released artifacts.
Store them on a binary repository (i.e., Nexus).
Use deployment automation to describe and manage internal and external dependencies.
Version your deployment automation scripts alongside your code.

Building in Quality and Security

Do not have separate quality and security phases after release.
Test for quality and security as part of your regular builds.
Integrate with the tools for the job.
SonarQube (quality).
WhiteSource (security/licenses).
Coverity Connect (security).
Create automatic reports and submit them to Jira.
Automate Jira ticket generation when security flaws or poor quality are found.