订阅内容

The power of correlating observability signals

Observability is a complex landscape to navigate for organizations. Many different signals can be collected, such as metrics and logs, and traces are becoming increasingly relevant to track applications. Over the past few years, correlating observability signals has become one of the most helpful ways to deal with the increasing volume of observability data, minimizing the time and effort required to troubleshoot clusters effectively.

In this case, correlating observability signals means following relationships to find related data in multiple heterogeneous stores. Despite the power of such correlation tools, no open source project focusing on Kubernetes clusters had yet been made available. That is why Korrel8r was created. Red Hat founded the project in 2022 and it was first introduced in KubeCon Europe 2023. It is currently an alpha project of the Red Hat Observability organization.

Benefits of Korrel8r

Korrel8r is a correlation engine for observability signals and observable resources. It can correlate multiple domains, diverse signals, inconsistent labeling and varied data stores.

What are the project goals?

Korrel8r can encode domain knowledge from site reliability engineers (SREs) and other experts as re-usable rules. These rules can automate navigation across observability signals and help users' focus on the data they need to diagnose cluster problems and their root causes.

Korrel8r rules describe relationships between signals. Given a start signal (or object), such as an alert in a cluster, and a goal, like finding related logs to a specific alert, the engine searches for goal data (in this case, logs) that is related to the start signal (in this case, a given alert) by some chain of rules.

Illustration of how Korrel8r connects data from different types of data store

Korrel8r aims to connect data from different types of data stores—even those characterized by different schemas—to reduce manual steps and minimize cluster troubleshooting efforts. Currently, Korrel8r connects to data stores such as Loki for logs, Prometheus/Thanos for metrics and the kube-API server for cluster resources. View the following architecture diagram or read the Korrel8r documentation for more information.

Illustration of korrel8r architecture

By providing a REST API, Korrel8r aims to be useful in various contexts. In the first application, a cluster console displaying resource or signal information will query Korrel8r to discover related resources and signals that can be displayed to the user. Korrel8r is not tied to the Red Hat OpenShift web console; it is a generic service that accepts queries identifying starting objects (resources or signals) and returns queries for related goal objects, possibly in different stores from the starting point. This correlation-as-a-service will be useful in other tools that analyze cluster observability data.

Dev preview: Access a first correlation experience in the Red Hat OpenShift web console

As announced a few weeks ago in the What’s new in Red Hat OpenShift Monitoring 4.14, Logging 5.8, and Distributed Tracing 2.9? blog, a first correlation experience is now available to OpenShift users directly in the web console as part of the latest Logging 5.8 z-stream release. In OpenShift, Korrel8r is officially named observability signal correlation for Red Hat OpenShift.

Korrel8r is available in the OpenShift web console if the Logs exploration UI is enabled. This dev preview feature aims to help you troubleshoot issues related to OpenShift clusters faster by taking you from a specific alert to its relevant log and from a particular log to its relevant metrics through links. The following image shows how, by deep diving into the Alert details information (starting point: Observe > Alerting UI > Alerts), you can use a new link: See related logs. Note that the button is not shown if correlated data is not found.

Screenshot of Alert details

Once you click the link, you are immediately directed to the Observe > Logs UI, providing the relevant log query. More information on accessing the Logging view in the OpenShift web console is provided here.

Screenshot of Logs UI

Similarly, once in the Logs UI, you can click on Metrics, which is provided in the Correlation column.

Screenshot of Metrics

When you click Metrics, Korrel8r will find the equivalent metric to that log and redirect you to the Observe > Metrics UI, as shown in the screenshot below.

Screenshot of Metrics graph

Look at the relevant documentation to learn more about it.

Deploying Korrel8r on an OpenShift cluster

You can install Korrel8r on an OpenShift cluster by using the Operator provided in the GitHub repository. Installing Korrel8r on an OpenShift cluster enables you to use the web console to view Korrel8r data.

Prerequisites

See this documentation for a complete description of the procedure.

What's next?

We are just beginning our observability signal correlation journey in OpenShift. The next steps include expanding the supported signals in Korrel8r—including Tempo—and providing OpenShift web console users with an improved user navigation experience by incorporating a debugging side-panel and a topology graph. Stay tuned for more! We look forward to receiving your feedback.


关于作者

Vanessa is a Senior Product Manager in the Observability group at Red Hat, focusing on both OpenShift Analytics and Observability UI. She is particularly interested in turning observability signals into answers. She loves to combine her passions: data and languages.

Read full bio

Background in distributed systems and messaging middleware, nowadays focused on observability in the cloud.

Read full bio
UI_Icon-Red_Hat-Close-A-Black-RGB

按频道浏览

automation icon

自动化

有关技术、团队和环境 IT 自动化的最新信息

AI icon

人工智能

平台更新使客户可以在任何地方运行人工智能工作负载

open hybrid cloud icon

开放混合云

了解我们如何利用混合云构建更灵活的未来

security icon

安全防护

有关我们如何跨环境和技术减少风险的最新信息

edge icon

边缘计算

简化边缘运维的平台更新

Infrastructure icon

基础架构

全球领先企业 Linux 平台的最新动态

application development icon

应用领域

我们针对最严峻的应用挑战的解决方案

Original series icon

原创节目

关于企业技术领域的创客和领导者们有趣的故事