5 software testing strategies to build into your CI/CD pipeline

Many agencies and enterprises are using continuous integration/continuous delivery (CI/CD) pipelines to automate the software delivery process. Yet once an organization establishes its main pipelines to orchestrate software testing and promotion, those pipelines often go unreviewed. The software promoted through the CI/CD toolchains frequently evolves, but the software release processes do not evolve. As an application changes, architects should consider whether it is effectively tested and kept up to date.

This article describes five testing strategies to consider incorporating into your pipeline. Regardless of your CI/CD implementation, introducing these testing strategies can improve the quality of your software releases.

1. API testing

Over the last couple of years, there's been an increasing focus on APIs and the value you can generate by leveraging them more extensively. An API is one of the most critical components of any application, working as the glue between different systems. The various entities that depend on the API are always expanding, ranging from mobile devices, Internet of Things (IoT) technology, human users, and applications.

Typically, APIs are expressed as RESTful- or SOAP-based endpoints, and you can interact with them universally, as long as API consumers and producers can agree on a contract. You expect the contract to remain consistent and provide expected results. Many popular tools easily plug into any CI/CD pipeline. Some of these tools include SoapUI, Swagger, and Red Hat's 3scale. These tools allow you to execute API tests directly from your pipeline, so you can check that your APIs are behaving as expected during the software release process.

2. GUI testing

Just like an API, the stability and functionality of graphical user interfaces (GUI) are critical to the success of an application's rollout. Suppose components of the GUI do not behave as expected, and you don't catch it before production rollout. In that case, problems become obvious because users can't access the app or parts of its functionality. It could become challenging to troubleshoot, given that these issues may reside in individual environments or browsers.

[ Learn about the Zero-Trust approach to designing security architectures. ]

By defining a robust GUI test suite that covers supported clients and browsers, you can catch these issues before users see them. GUI testing tools simulate the points and clicks that a human user would use from their computer to execute a sequence of actions and then compare expected results to the actual results.

Selenium and Appium are two popular GUI testing tools, and both allow testers to develop sequences of steps that a user would walk through using the GUI. Both tools can participate in any CI/CD pipeline. By incorporating these tests as part of the automated release cycle, you can have greater confidence that the GUI functions as expected across various platforms and browsers and that components of the GUI have not broken during changes to server-side application logic.

[ Download the checklist: Considerations for implementing DevSecOps practices. ]

3. Nonfunctional testing

Do you ever wish there were a way to prepare your teams better for unscheduled outages and failures? Does the thought of unexpected traffic spikes give you anxiety about not knowing how the system will behave? Perhaps it's time to give attention to nonfunctional testing tools.

Nonfunctional testing is a broad subcategory of testing that incorporates how the application behaves under external and often uncontrollable factors, such as load, stress, volume, as well as unexpected and infrequent environmental events or failures (network disconnectivity, upgrades, and more).

You often encounter issues around nonfunctional aspects once an application is in production. Environmental triggers, such as a network interruption in the datacenter or critical business dates that could cause unusual spikes in traffic, are often outlying situations that can lead to a crisis if an application cannot handle these events with grace. These events are usually quite difficult to prepare for and recover from when they are in process.

To help prepare for unforeseen events and understand how the application will perform during traffic spikes or environment failures, incorporate gates for nonfunctional testing within your pipeline before an application gets released to the production environment. This is a broad category, so you have many tools for each domain that also plug into CI/CD pipelines.

[ Don't try to recreate what was normal before the pandemic. Learn from leading CIOs in a new report from Harvard Business Review Analytic Services: Maintaining Momentum on Digital Transformation. ]

4. AppSec testing

Security isn't addressed in many enterprises until later in the application release cycle. With the introduction of DevSecOps, there is a greater focus on incorporating security checkpoints throughout the application release lifecycle, not just left as an afterthought. Incorporating security scanning as part of the CI/CD pipeline, you can consistently enforce security scanning using the automation in place. The earlier you encounter security vulnerabilities, the cheaper they are to resolve. There are many types of security scanning tools in the market today. Many mature teams leverage different security scanning tools depending on the types of tested assets and the type of scanning that makes sense.

Static Application Security Testing (SAST) focuses on analyzing code in source control. SonarCube and Veracode are tools to integrate into your pipeline to identify potential code vulnerabilities early in the development cycle.

Dynamic Application Security Testing (DAST) tools analyze applications in their execution state. This helps find vulnerabilities that only present themselves when the application runs, such as SQL injection or code execution attacks.

Container scanning technology has gained popularity over the last couple of years as more teams leverage Linux containers for their production workloads. These tools introspect the container images to uncover issues within the content that makes up the container. As containers have different layers, each containing their own packages and configuration, it's important to have a platform for verifying that the contents of a container are safe for promotion. Common platforms for container scanning include Red Hat's Stackrox, Aqua, and Blackduck.

The more comprehensive your pipeline's approach to security scanning, the greater your overall security posture is within your organization. Introducing checkpoints early and often is a great way to impact released software's quality.

5. Regression testing

It is not unusual for changes to one component to have downstream effects across full system functionality. Since modern-day software involves many interconnected parts, it's important to establish a solid regression testing strategy.

Regression testing should be part of any application release. It should focus on verifying that existing and business functionality performs as expected regardless of changes made to the system. Without regression testing in place, bugs and vulnerabilities can appear in components of the system that the release may have never changed. These problems become harder to identify and diagnose once the application is released. Teams doing troubleshooting may not know where to begin, especially if the release did not modify the failing component.

When devising a regression testing strategy, you can break it into three different phases:

1. QA testers generate tests that cover end-to-end functionality. You can use Selenium, Appium, and Cucumber to create these test suites.
2. Decide on when and how to execute the tests. This is the phase where you could incorporate regression testing into the CI/CD pipeline.
3. As the application evolves, maintenance of the existing test suite becomes more important.

Tools like Healenium help maintenance engineers keep tests updated as the application evolves.

Regular testing keeps software up to date

Automated software delivery is not a new concept for many agencies and organizations. Once an enterprise establishes its main pipelines to orchestrate the testing and promotion of software, architects rarely return to revisit and check it is running smoothly. The software you promote through your CI/CD toolchains often evolves, but the software release processes do not. As applications change, architects need to incorporate thorough and effective testing that is up to date with the industry's evolution.