Red Hat reduced software release effort by 87%

The Red Hat Delivery Acceleration team has been working with Red Hat’s development teams to address their pain points. Teams needed to release code faster, which meant less time to gather all the facts to support deployment decisions manually. Delivery Acceleration set about building a release automation pipeline based on tooling native to Red Hat OpenShift Platform Plus and affiliated products. The automation has cut manual steps by 75% and time spent on releasing software by 87%. The centralized automation means all 42 IT development teams can release software consistently and without business disruption at the same accelerated velocity.

Benefits: 

• Cut development team time spent on software releases by 87%.
• Accelerated issue resolution and safeguarded compliance.
• Reduced training requirements and avoided business disruption.

Eliminating laborious manual tasks is a clear benefit to developers. Whether they’re implementing feature requests or building something new, they have a significant volume of (digital) paperwork to fill in before their code can be released, and that hinders the rapid delivery that companies are trying to achieve.

“Modern software teams must deploy their code faster and meet ever-increasing compliance requirements,” said Sherry Gentry-Gasper, Senior Director of Delivery Acceleration, Red Hat. “But when releases occur more frequently, there’s less time to manually gather all the facts to support deployment decisions and meet audit requirements.”

What’s more, providing senior DevOps skills for all internal development teams—whether they’re working on business applications or Software-as-a-Service (SaaS) products—is often not viable. “Here at Red Hat, the software delivery lifecycle was made up of a series of siloed steps,” said Derek Haynes, Senior Principal Software Engineer, Red Hat. “It lacked end-to-end orchestration.”

The Delivery Acceleration team, as it’s known internally at Red Hat, is focused on providing Red Hat’s developers across the broader business with reusable automated pipelines that follow continuous integration and continuous delivery (CI/CD) best practices. As a crucial part of DevOps, CI/CD aims to streamline and accelerate the software development lifecycle by automating tedious and error-prone manual processes. It automates manual tasks to improve efficiency, speed up software delivery, reduce errors, and ensure compliance.

“We want to eliminate non-value-added tasks so that developers can focus on what they do best,” said Gentry-Gasper. “And that’s solving business process problems by developing applications to run on Red Hat OpenShift.”

The 7-person Delivery Acceleration team began by listening to the thoughts of the various delivery teams around their biggest pain points, particularly those relating to releases and compliance. “The teams told us that they needed to deploy their code faster, safer, and with better quality to maximize value and customer experience, but that meant less time to manually gather all the facts to support deployment decisions,” said Haynes.

Based on what the delivery teams said, the Delivery Acceleration team set its first mission: to build release automation for Red Hat’s Core Business Platforms (CBP) team. Release automation simplifies decision making, increases consistency, and allows quality releases to be deployed efficiently and with minimal risk.

The team examined the tooling natively available within Red Hat OpenShift and affiliated products and then built the automation based on those. “We strung the various tools together to optimize the, efficiency of the release process,” said Haynes.

Those tools include:

• GitLab for version control
• ServiceNow for incident and request management
• Nexus repository for common libraries
• JMeter for performance testing
• Jira for issue and project tracking
• Polarian for automated test management
• SonarQube for static code analysis
• Selenium for UI test automation

The automated pipeline runs on a managed Red Hat OpenShift Platform Plus foundation, which also includes:

• SignalFx and Splunk telemetry
• Helm templates
• Red Hat Quay image repository
• HashiCorp Vault for storing credentials
• ArgoCD to manage deployments of applications in OpenShift

An internal warehouse based on Snowflake captures information for compliance reporting with Kafka messaging and workflow orchestration, sending the required information to Snowflake. The overall pipeline automates compliance through compliance as code, so developers don’t need to spend time ensuring that compliance needs are met consistently by all development teams.

Cut delivery effort by 87%, increasing delivery velocity

“As an engineer, OpenShift simplified the development process for creating pipelines,” said Ravelle Kelley, Principal Software Engineer, Delivery Acceleration, Red Hat. “Developing a secure CI/CD process was straightforward. Altogether, the automation allows us to provide consistent release logs and metadata, streamlining release and compliance processes, eliminating duplication, and saving time.”

Specifically, removing manual touchpoints and introducing automated compliance controls (ACC) and a user acceptance testing (UAT) platform has accelerated the speed at which teams deliver value to the business. “Release automation has freed up developers to focus on what they do best and increased their delivery velocity,” said Gentry-Gasper.

Introducing automation and cutting the number of manual steps from 12 to 3 for every CBP release redhat.com Case study Red Hat automation accelerates software delivery across 42 teams has cut the number of hours spent on delivery each month from 215 to 28—by 87%, which is significant. Moreover, the new automated pipeline has reduced integration complexity, cutting integration effort from 2 engineers over 8 weeks to 1 engineer over a fortnight.

Accelerated problem-solving and safeguarded compliance

The new pipeline also assures consistency across many aspects of development. “As an example, the automation creates the Splunk logs, meaning all Splunk logs are consistent,” said Gentry-Gasper. “We have teams running jobs or integrations that impact different platforms. They can look at those logs themselves to understand where the issue is because they are consistent. They don’t have to turn to someone running the platform for an explanation.”

Consistency not only speeds up issue resolution but also safeguards compliance. “Migrating to the new release automation pipelines allowed my team to adopt a modern GitOps deployment model, aligned with industry best practices,” said Vidhya Chidambaram, Principal Software Engineer, Partner Integration Team, Red Hat. “JIRA-initiated releases, integrated consistently with ServiceNow, helped accelerate our release process while ensuring compliance with internal governance and audit requirements.”

From a compliance perspective, automation guarantees that vital compliance controls are never missed, mitigating any risk of non-compliance. “Compliance is built into the automation so developers can be confident that all requirements are taken care of and all the evidence auditors require is produced automatically,” said Gentry-Gasper. “They no longer need to deal with those time-consuming manual steps because the automation takes care of them.”

Reduced training requirements while avoiding business disruption

Red Hat no longer needs to train someone in every one of its 42 information technology development teams in DevOps best practices; that expertise only needs to sit within the Delivery Acceleration team. Previously, only development teams that were able to train or onboard their own DevOps specialists could build automation into their software development lifecycle.

The centralized expertise means every development team can access that expertise and the optimal automation built by the Delivery Acceleration team, allowing them to focus on what they do best: building value for the business.

All teams now release software at a similar velocity and with minimal disruption to the business. “Teams are now releasing software at a similar velocity, which is less disruptive for the business,” said Haynes. “Previously, some teams had their own DevOps expertise and, with that, had fully automated their software releases. They could release their software into production every day. Other teams were less mature and only released sporadically.” Having consistent release schedules across development teams makes life easier for the business, too; business teams can expect a consistent experience no matter which development team they are engaging with.

Equally importantly, automated releases bring less disruption. The outages the business had to manage when some teams released software are a thing of the past.

Red Hat’s Delivery Acceleration team still spends time with development teams to understand their current state, pain points, and potential areas for improvement. Having delivered Red Hat’s first end-to-end release automation pipeline for applications running on Red Hat OpenShift, the team now plans to extend the pipeline to enterprise business applications that Red Hat uses internally, including Salesforce, Workday, and SAP.

“Many enterprise applications are heavily connected, so providing releases at a consistent pace is very important for the business,” said Haynes. “And since many of those vital enterprise applications don’t support CI/CD pipelines, the automation we’ve built will allow developers to release more frequently and consistently, minimizing business disruption.”

The team is also focused on providing tooling recommendations, documentation, and enablement to scale pipelines. “We have led by example as an empowered agile team working in an environment of trust, transparency, and fun while partnering with development teams to bring about a continuous and automated transition from strategy to delivery,” said Gentry-Gasper. “The Delivery Acceleration team is helping to ensure Red Hat’s future success.”