How to approach DevOps metrics

DevOps metrics track the effectiveness of DevOps practices, which relate to software development and IT operations. DevOps aims to deliver software more quickly, reliably, and stably. It prioritizes self-service tools, automation, and communication, with an emphasis on developer productivity and satisfaction.

If you’ve invested in DevOps, you’ll want a consistent set of metrics to monitor your progress. Your preferred approach will depend on what’s important to your organization. Do you want to measure software output or the health of your teams? This article examines 2 of the most popular frameworks: DORA (which focuses on software) and SPACE (which focuses on teams).

DevOps centers around a culture of continuous improvement. Deciding what you’re trying to improve—and how you’ll measure it—is an important strategic step. Two influential frameworks for DevOps metrics are DORA and SPACE. 

Because the primary goal of DevOps is to speed up the release of working software, you might prioritize measurements related to that goal. DORA (which is short for DevOps research and assessment) metrics fit this case.

But there’s more to DevOps than how much working software you ship. You might focus on qualities like team satisfaction and collaboration. SPACE (which stands for satisfaction, performance, activity, communication and collaboration, and efficiency) emphasizes team-related metrics.

These distinct approaches to measuring DevOps reflect different philosophies about what’s important to measure. But they don’t reflect a binary choice. Both frameworks can complement each other and measure different parts of your DevOps goals. 

DORA

DORA metrics come out of the DORA research program, which is supported by Google Cloud. In 2020 the DORA program established the ”Four Keys” for measuring the performance of a software development team:

• Deployment frequency, or how often an organization successfully releases software to production. This requires the team to define what counts as a successful deployment. Once you decide, you can make a calculation, such as the median number of days per week with at least 1 successful deployment.
• Lead time for changes, or the time it takes a commit to get into production. This requires tracking when a commit happened and when the deployment happened.
• Change failure rate, or the percentage of deployments causing a failure in production. To calculate this, you need to know how many deployments you’ve made and how many bugs or incidents were reported, such as in an issue-management system.
• Time to restore service, or the time it takes to recover from a failure in production. You’ll need to track creation dates and resolution dates for each incident—something you can pull from an incident-management system.

A 5th key metric was added the following year:

• Reliability, which encompasses availability, latency, performance, and scalability. The DORA framework measures an organization’s ability to meet or exceed its reliability targets.

The focus of this research and set of metrics is better business outcomes. Teams can track their software performance by pulling data from their software pipelines into a dashboard and monitoring their progress over time. The DORA framework also assigns performance ratings (elite, high, medium, and low) based on the levels of each metric.

A major advantage of the DORA framework is that it’s based on readily available measurements from tools software teams already use, like GitHub, GitLab, and incident-management systems. You’ll need to pull the metrics into dashboards, but the data’s already there.

SPACE

The SPACE framework for DevOps metrics attempts to capture different dimensions of productivity. A team of researchers introduced SPACE in a 2021 article. They argued that engineering work is too complex to capture in a single dimension or metric. Software development requires trade-offs, and teams need a holistic approach that includes more elements of their work.

The SPACE framework includes 5 distinct dimensions to measure:

• Satisfaction and well-being, which seek to measure how developers feel about their work, and how healthy and happy they are. This can require gathering data through surveys about employee satisfaction, whether developers have access to tools and resources they need, and levels of burnout caused by workplace stress.
• Performance, or the outcome of a system or process (as distinct from the output). This is more than just the amount of code or the business impact it has. Performance is measured by quality—such as the reliability of the code or ongoing service health—and impact—for example, customer satisfaction or customer adoption and retention.
• Activity, which tracks actions or outputs completed in the course of performing work. You can capture developer activity levels in many ways, such as the volume of commits or release counts, but SPACE framework emphasizes not to use these in isolation.
• Communication and collaboration, which together capture how people and teams share information and work together. Some metrics that can help measure communication, collaboration, and coordination include discovery of documentation and expertise, how quickly work is integrated, quality of work reviews, network metrics that show who is connected, and onboarding time for new members.
• Efficiency and flow, which measure the ability to complete work or make progress with limited interruption or delays. SPACE suggests measuring efficiency and flow through metrics such as number of handoffs in a process, perceived ability to stay in flow and complete work, interruptions, and time measures through a system.

You can measure and track each of the SPACE metrics at the individual, team, or system level.

The SPACE framework tries to expand the way teams think about DevOps, tilting away from measuring raw output toward a holistic view of productivity. It may require creating new processes such as surveys to gather metrics that don’t already exist, such as developer satisfaction levels.

The DevOps metrics you choose to use should reflect your unique priorities and be based on data you can realistically measure. Some of your metrics for DevOps success could focus on outputs, including velocity and quality of software delivered, as in the DORA framework. Or they could focus on the complex trade-offs software developers must make to be more productive, as in the SPACE framework.

Questions you can ask that will help you choose what to measure include:

• What are your organizational goals? Your goals may focus on improving velocity, or reliability and resilience. Or you could be more interested in a holistic developer experience, including developer satisfaction, ease of collaboration, and efficiency.
• What data can you collect? Your continuous integration and continuous deployment (CI/CD) pipelines may be your best source of data about how software changes are delivered. But you also might need to employ surveys to gather qualitative data.
• Are your automation and communication systems working effectively? To capture DevOps metrics, you need visibility into bottlenecks, communication across team functions, and how decisions are made. Making sure your DevOps systems work as designed may be a 1st step before you can get reliable data about the factors you want to measure.

DevOps is related to platform engineering, which addresses internal platforms and tools that support DevOps workflows. Some metrics that assess DevOps performance are drawn from platform engineering tools. The discipline of developer experience, or DevEx, is also reflected in DevOps metrics. 

Platform engineering seeks to enhance the developer experience by easing burdens on development teams. It includes providing an internal developer platform (IDP), which establishes a set of common, reusable tools and capabilities for developers. Platform engineers are responsible for designing and maintaining golden paths, which are well-documented, supported ways to build and deploy software according to the organization’s standards.

The choices that inform the IDP, golden paths, and other related artifacts are strongly related to DevOps metrics. As you’re establishing the processes to make DevOps effective, you’re also setting up what matters and what you can realistically measure. For example, CI/CD pipeline will be your source of visibility into software commits and releases.

Red Hat has platforms, tools, and consulting services that can help you measure your DevOps productivity.

Red Hat OpenShift

The platform where you run your applications will influence the success of your DevOps metrics strategy. Red Hat® OpenShift® provides a consistent platform across cloud, on-premise, and edge environments. It supports CI/CD workflows through the Red Hat OpenShift GitOps and Red Hat OpenShift Pipelines operators. And Red Hat OpenShift observability gives you a view of the performance and health of your applications.

Red Hat Advanced Developer Suite

In alignment with DevOps’ focus on enhanced productivity, Red Hat Advanced Developer Suite extends the capabilities of Red Hat OpenShift by providing the essential components to speed up developer productivity, strengthen software supply chain security, and support AI-powered development.

Red Hat Developer Hub

Included in Red Hat Advanced Developer Suite, Red Hat Developer Hub is an internal developer portal. It visually consolidates developer tooling and services—from service catalogs and documentation to CI/CD pipelines—into a single, navigable hub, dramatically improving developer experience and organizational productivity.

Red Hat Consulting

Technology experts from Red Hat Consulting can support your DevOps programs with hands-on collaboration. With techniques such as service blueprints and metrics-based process mapping (MBPM), consultants can help you better understand and measure your workflows.