Constructing a DevOps pipeline is an essential part of a software architect's process when working in a software engineering team. In the past, as I participated as a technical interviewer at Red Hat, I was quite surprised to find very few people could clearly describe what a DevOps pipeline is and what a Continuous Integration and Continuous Deployment (CI/CD) pipeline is. You will understand what these pipelines are by reading this article.
If you prefer video, check out my summary of this research below.
Let's have a little discussion about what DevOps is before jumping into the DevOps pipeline. DevOps, by itself, is technical not defined. But DevOps, in common practice, is a software development methodology often associated with specific practices and tools that help implement those practices. It is mainly focused on increasing the frequency of software deliveries to build more resilient systems. You can think of DevOps like other software development processes such as Agile, Lean, Scrum, or Waterfall.
[ For more on DevOps, read What is DevOps Anyway?]
One of the most approachable books written about the DevOps process is the Phoenix Project by Gene Kim. A reference page found in the back of the book explains the software deploy frequencies of different companies. It states that a typical enterprise releases software only once every nine months. In contrast, Netflix, for example, releases software updates thousands of times per day due to its well-constructed DevOps process.
* Image source - O'Reilly
More releases without significant change to team processes can lead to less resiliency rather than more. DevOps processes that only focuses on the frequent delivery aspect can be misleading and undesirable. A DevOps team also needs to emphasize the reliability, safety, observability, and resiliency aspects of the system. These aspects are positively affected by more frequent releases, as discussed in Accelerate by Nicole Forsgren, Ph.D., and co-authors.
Alright, so we now understand what the DevOps process is, but what does DevOps promise to deliver? To understand this, we need to understand what the DevOps pipeline is.
A DevOps pipeline is the bread-and-butter of the DevOps process. The term is used most commonly to discuss the tools, processes, and automation frameworks used to build software artifacts. The most well-known DevOps pipeline that kicked off the DevOps trend is Jenkins, which is an open source tool built in the Java programming language. Now, we have many DevOps pipeline tools like Argo, Harness, GitHub Actions, and Travis CI.
A DevOps pipeline are important to understand based on the organization that will use them. These teams can be largely broken down into two aspects: Infrastructure (Operations) and Code Development (Development). Whichever role you belong to depends on a number of different factors–including the existing DevOps pipeline, the size of your company, your position at the company–but is most connected to you responsibility to either produce software (Development) or manage where software is deployed (Operations).
For example, imagine that your company is a startup that only has a few engineering team members. The company has no previously-designed DevOps pipeline. As an architect or a team lead, your responsibility is to create an elegant solution that improves every software delivery aspect. You need to come up with at least the Minimum Viable Product (MVP) pipeline that delivers the infrastructure aspect as well as the code deployment aspect.
If you work in a large enterprise with an existing software delivery methodology, you will find the need to focus on both the infrastructure side as an operations engineer and the code delivery side as an application developer. That need to span this gap is the true goal of DevOps. Understanding what stage your team's DevOps process is at, your role, and how big your team is helps you determine what you need to focus on when you join an engineering team.
Now we finally get to the exciting part: Learning how to design a DevOps pipeline. First, we will cover the infrastructure side, and then move on to the code development side.
DevOps pipeline for infrastructure
As an infrastructure engineer—also called an operations engineer (Ops) or a DevOps engineer—you are responsible for building the environment necessary to host and run applications. Most of the time, this means a cloud environment, which can be either running on top of a virtual machine (VM) or in containers. Please see the architecture diagram below, which illustrates a general DevOps pipeline process applicable to an infrastructure engineer or an architect.
The high-level step-by-step process is listed below:
- A DevOps pipeline job is triggered.
- The pipeline job executes a task to check out the code from a Source Control Management (SCM), like GitHub or GitLab. This SCM includes the DevOps pipeline script and configuration management tools (e.g., Ansible or Bash) to be executed by the DevOps pipeline job steps.
- The DevOps pipeline authenticates to an SCM and checks out the code.
- The DevOps pipeline executes the job based on the steps outlined by the DevOps pipeline.
- To build a cluster or perform any infrastructure-related activity, an ephemeral pipeline agent is usually needed. A DevOps pipeline authenticates with an image registry (e.g., Quay, Artifactory, DockerHub) and pulls an image that is used for spinning out the pipeline agent.
- A temporary pipeline agent, which usually already has the scripts, libraries, and tools (another use for Ansible) necessary to execute the scripts from the SCM, spins out and performs actions based on the DevOps pipeline script's instructions.
- If the requested infrastructure pipeline action is about building or customizing an image, the temporary pipeline agent may pull the additional images from the image registry or push them to the image registry. The pipeline agent shuts down after the execution.
- If the requested infrastructure pipeline action is about creating, updating, or deleting a cluster, the temporary pipeline agent may authenticate and then perform the infrastructure operation activity. The pipeline agent shuts down after the execution.
- After a pipeline action completes, the pipeline notifies the engineering team or stakeholders of the result, either through an email or another messaging platform like Element, Microsoft Teams, or Slack.
Infrastructure DevOps pipeline technology stacks
Let's explore some commonly-used technology stacks in the infrastructure DevOps pipeline.
First, here are a few examples of the DevOps pipeline tools. Note that these tools are usually equivalent to the DevOps pipeline for the application developers as well:
- GitHub Action
You also have several choices for the Source Control Management (SCM) tools. Here are a few examples:
If you want to store operating system images or container images, you need some type of image registry. Here are a few options that you can explore:
Most importantly, to perform the infrastructure-related activities, such as provisioning a cluster, building a custom VM image, etc., you need to have configuration management tools and shell scripts, such as the ones below:
- Ansible: This configuration management tool is excellent at installing packages, creating a VM image, and carrying out other VM-related operations.
- Bash script: While it takes more work to make idempotent, Bash can be useful for all types of operations tasks.
Lastly, you need a cloud provider or an environment to deploy and manage your virtual machine, a Kubernetes container, or a cluster. Any public cloud provider will work, but you can start with a virtual machine or a container environment running on your local computer.
Most common infrastructure DevOps activities
Although it is impossible to list all infrastructure activities that can be performed by the DevOps pipeline, here is a list of some of the most commonly-used infrastructure tasks. and tools that are ideal for the task:
- Provisioning a cluster (oc or kubectl)
- Deleting a cluster (oc or kubectl)
- Making a change in a cluster (oc or kubectl)
- Building a container image (docker or podman)
- Building a VM image (Ansible)
- Installing packages and performing post-installation activities (Ansible)
- Making a change to a network in a cluster (Cluster Network Operator)
- Provisioning storage in a cluster (oc for dynamic provisioning)
- Backing up the state of a running environment (Operators like this one)
- Restoring a previously-working state of a running environment (possible on OpenShift)
DevOps pipeline for application developers
Let's switch gears slightly to explain how the DevOps pipeline for application developers works. If the DevOps pipeline for the infrastructure team is centered around how to provide a stable environment, the DevOps pipeline for the application developers is designed to deliver software updates as often as possible. (Remember from earlier: more frequent update leads to greater stability according to research.)
Here is a possible architecture diagram illustrating the DevOps pipeline for the application developers. Note that this is a simplified diagram in a non-container context. With the container-based approach, additional steps can be introduced.
The high-level step-by-step process is listed below:
- A DevOps pipeline job is triggered.
- The pipeline job executes a task to check out the code from a Source Control Management (SCM), like GitHub, BitBucket, or GitLab. This SCM includes the DevOps pipeline script and the source files for a specific programming runtime (e.g., Java, Python, Go, etc.). If the software building process involves deployment to a container/Kubernetes environment, the SCM repo may include the Dockerfile or Source-to-Image (S2i) configuration file.
- The codes get checked out, and the DevOps pipeline script and the necessary source codes are now inside the DevOps pipeline's temporary workspace.
- The pipeline executes the step-by-step tasks based on the DevOps pipeline scripts. Depending on the programming runtime, this usually invokes build scripts using tooling such as Maven, Grunt, or Makefile.
- In addition to the unit test, the pipeline runs additional end-to-end tests and other quality confirmations, and then publishes the result through a tool like SonarQube. If the build fails or succeeds, the pipeline should also notify the developer team about the result.
- If the application is successfully built and passes all the tests, the packaged application can be deployed to a virtual machine or a container in a Kubernetes environment.
Application developer DevOps pipeline technology stacks
Many tools used by the application developer-focused pipeline are similar to those of the infrastructure-focused pipeline. One addition is a code quality tool. A few examples of the code quality tools are shown below:
To build software, you need a build tool. The recommended tool differs based on the programming runtime. Here is a table showing some commonly-suggested build tools.
|Programming runtime:||Build tools:|
|Java||Apache Maven, Apache Ant, Apache Gradle|
|Python||Fabricate, Pynt, Fabric|
Most common application development DevOps activities
Here are a few common application development DevOps tasks that can be performed by the pipeline:
- Compile and build software source code.
- Run the tests and generate test results.
- Deploy the code package to a virtual machine or a container.
- Merge the code back to an SCM branch.
- Close a User Story or a task in an Agile tool.
The idea of separating the Continuous Integration (CI) process from the Continuous Delivery (CD) process has not been a common practice adopted by the software industry until recently. The most well-known DevOps pipeline tool, Jenkins, started when the distinction between CI and CD was vague. As the software engineering process evolved, the benefits of separating the CI process from the CD process have become apparent.
Here is a table that defines the difference between Continuous Integration, Continuous Delivery, and Continuous Development.
|Process:||What it is:|
|Continuous Integration||Developers merge their changes back to the main branch as often as possible. The developer's changes are validated by creating a build and running automated tests against the build.|
|Continuous Delivery||Releases new changes to one or more environments, including production. This means that on top of having automated testing, you also have an automated release process, and you can deploy your application at any point in time by clicking on a button.|
|Continuous Development||Every change that passes all stages of your production pipeline is released to production. There's no human intervention, and only a failed test will prevent a new change to be deployed to production.|
While a team can use the Jenkins pipeline to implement the entire CI/CD process, the industry trend now leans toward adopting a separate DevOps pipeline tool for the CD process. For example, while one company may still use Jenkins to build a software or build a virtual machine image, Harness or Argo delivers the package to a cluster or a virtual machine environment.
Putting the pipeline together
In this article, you learned about the heart of DevOps and DevOps pipelines. You also learned how to create DevOps pipelines from two different perspectives: One with infrastructure engineers in mind and one for the application developers. You also saw the difference between Continuous Integration (CI) and Continuous Delivery (CD) processes and how the industry trend is to separate the two. When applied to both operations and application development, these technologies lead to more deploys per day, which has been shown by research to lead to more resilient systems. That's why DevOps is powerful.