When an operator fails, it can mean loss of critical functionality. When an admin sees an operator at risk, it must be fixed before it becomes a critical issue. As an operator developer, you want to help your users get this valuable info, fast. Here's how the Red Hat OpenShift Virtualization operator team created an Operator health metric that can help.
Currently, operators have different metrics and alerts that are unique to their needs and reflect their conditions. Yet there is no aggregation that can tell the user, through one single pane of glass, what the operator's general health is. Moreover, it’s not clear to users which alerts impact the operator's health and which are related to workloads.
In the OpenShift Virtualization operator, we created a metric that performs this aggregation and enables us to clearly define the operator's health.
To achieve this, we added three labels to each of the operator alerts:
1. kubernetes_operator_part_of
– operator name
2. kubernetes_operator_component
– operator component (can be the same as the above)
3. operator_health_impact
– indicates the impact of the issue on the operator's health. This label will get one of the following values:
critical
indicates an issue that negatively impacts the operator's health, and the user should take action immediately to solve it.warning
indicates an issue that might soon negatively impact the operator's health, and the user should solve the issue.none
indicates an issue that doesn't affect the operator's health. In most cases, this is related to the workload rather than the operator itself.
We also created and added a new metric that aggregates the metrics conditions by the name kubevirt_hco_system_health_status
.
Its value is calculated by the following:
Degraded
=True
or !Available
=True
->kubevirt_hco_system_health_status
= 2 (Error)Progressing
=True
or !ReconcileComplete
=True
->kubevirt_hco_system_health_status
= 1 (Warning)- Else (
Available
=True
andDegraded
=False
andProgressing
=False
andReconcileComplete
=True
) ->kubevirt_hco_system_health_status
= 0 (Healthy)
We created a recording rule for the health metric called kubevirt_hyperconverged_operator_health_status that will indicate the overall health of our operator, link. Values: healthy (0), warning (1) or critical (2).
It is based on both alerts and operator conditions. Its value will be the minimum of the two mentioned above.
This metric provides a clear operator health status and enables you to easily detect the causes of the issue by checking the firing alerts and their health impact.
Health metrics based on Conditions vs. Alerts
Conditions-based health metric | Alerts-based health metric |
Doesn’t necessarily indicate a real issue since issues can be resolved by k8s. | Usually indicates a real issue since there is an evaluation time prior to each alert. |
Harder to track the issue and sub-operator since data in the conditions is aggregative. | Easy to track what the issue is since we can see the alerts. |
Precision depends on the code coverage. | Precision depends on the alerts coverage. |
Doesn’t require additional code changes. | Requires adding the the labels: - operator_health_impact to differentiate between operator and workload alerts to determine if they impact the health of the operator.- kubernetes_operator_part_of - To identify the alerts related to the specific operator. |
It is the convention for reporting health status in OCP operators. | Not the convention. The community is not sure why this approach would be better. |
About the author
Shirly has been with Red Hat since January 2014 and currently serves as the OpenShift Virtualization Observability Team Lead. In this role, Shirly concentrates on improving the operator observability within the Kubernetes and OpenShift ecosystem, to ensure it effectively addresses user needs. Over the years, Shirly has been involved in numerous projects, driven by a genuine passion for enhancing system efficiency and user experience. She is also proactive in sharing the knowledge and best practices garnered with the broader operators community.
More like this
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit