The power of correlating observability signals

Observability is a complex landscape to navigate for organizations. Many different signals can be collected, such as metrics and logs, and traces are becoming increasingly relevant to track applications. Over the past few years, correlating observability signals has become one of the most helpful ways to deal with the increasing volume of observability data, minimizing the time and effort required to troubleshoot clusters effectively.

In this case, correlating observability signals means following relationships to find related data in multiple heterogeneous stores. Despite the power of such correlation tools, no open source project focusing on Kubernetes clusters had yet been made available. That is why Korrel8r was created. Red Hat founded the project in 2022 and it was first introduced in KubeCon Europe 2023. It is currently an alpha project of the Red Hat Observability organization.

Benefits of Korrel8r

Korrel8r is a correlation engine for observability signals and observable resources. It can correlate multiple domains, diverse signals, inconsistent labeling and varied data stores.

What are the project goals?

Korrel8r can encode domain knowledge from site reliability engineers (SREs) and other experts as re-usable rules. These rules can automate navigation across observability signals and help users' focus on the data they need to diagnose cluster problems and their root causes.

Korrel8r rules describe relationships between signals. Given a start signal (or object), such as an alert in a cluster, and a goal, like finding related logs to a specific alert, the engine searches for goal data (in this case, logs) that is related to the start signal (in this case, a given alert) by some chain of rules.

Illustration of how Korrel8r connects data from different types of data store

Korrel8r aims to connect data from different types of data stores—even those characterized by different schemas—to reduce manual steps and minimize cluster troubleshooting efforts. Currently, Korrel8r connects to data stores such as Loki for logs, Prometheus/Thanos for metrics and the kube-API server for cluster resources. View the following architecture diagram or read the Korrel8r documentation for more information.

Illustration of korrel8r architecture

By providing a REST API, Korrel8r aims to be useful in various contexts. In the first application, a cluster console displaying resource or signal information will query Korrel8r to discover related resources and signals that can be displayed to the user. Korrel8r is not tied to the Red Hat OpenShift web console; it is a generic service that accepts queries identifying starting objects (resources or signals) and returns queries for related goal objects, possibly in different stores from the starting point. This correlation-as-a-service will be useful in other tools that analyze cluster observability data.

Dev preview: Access a first correlation experience in the Red Hat OpenShift web console

As announced a few weeks ago in the What’s new in Red Hat OpenShift Monitoring 4.14, Logging 5.8, and Distributed Tracing 2.9? blog, a first correlation experience is now available to OpenShift users directly in the web console as part of the latest Logging 5.8 z-stream release. In OpenShift, Korrel8r is officially named observability signal correlation for Red Hat OpenShift.

Korrel8r is available in the OpenShift web console if the Logs exploration UI is enabled. This dev preview feature aims to help you troubleshoot issues related to OpenShift clusters faster by taking you from a specific alert to its relevant log and from a particular log to its relevant metrics through links. The following image shows how, by deep diving into the Alert details information (starting point: Observe > Alerting UI > Alerts), you can use a new link: See related logs. Note that the button is not shown if correlated data is not found.

Screenshot of Alert details

Once you click the link, you are immediately directed to the Observe > Logs UI, providing the relevant log query. More information on accessing the Logging view in the OpenShift web console is provided here.

Screenshot of Logs UI

Similarly, once in the Logs UI, you can click on Metrics, which is provided in the Correlation column.

Screenshot of Metrics

When you click Metrics, Korrel8r will find the equivalent metric to that log and redirect you to the Observe > Metrics UI, as shown in the screenshot below.

Screenshot of Metrics graph

Look at the relevant documentation to learn more about it.

Deploying Korrel8r on an OpenShift cluster

You can install Korrel8r on an OpenShift cluster by using the Operator provided in the GitHub repository. Installing Korrel8r on an OpenShift cluster enables you to use the web console to view Korrel8r data.

Prerequisites

See this documentation for a complete description of the procedure.

What's next?

We are just beginning our observability signal correlation journey in OpenShift. The next steps include expanding the supported signals in Korrel8r—including Tempo—and providing OpenShift web console users with an improved user navigation experience by incorporating a debugging side-panel and a topology graph. Stay tuned for more! We look forward to receiving your feedback.


執筆者紹介

Vanessa is a Senior Product Manager in the Observability group at Red Hat, focusing on both OpenShift Analytics and Observability UI. She is particularly interested in turning observability signals into answers. She loves to combine her passions: data and languages.

Read full bio

Background in distributed systems and messaging middleware, nowadays focused on observability in the cloud.

Read full bio