An operators health metric

When an operator fails, it can mean loss of critical functionality. When an admin sees an operator at risk, it must be fixed before it becomes a critical issue. As an operator developer, you want to help your users get this valuable info, fast. Here's how the Red Hat OpenShift Virtualization operator team created an Operator health metric that can help.

Currently, operators have different metrics and alerts that are unique to their needs and reflect their conditions. Yet there is no aggregation that can tell the user, through one single pane of glass, what the operator's general health is. Moreover, it’s not clear to users which alerts impact the operator's health and which are related to workloads.

In the OpenShift Virtualization operator, we created a metric that performs this aggregation and enables us to clearly define the operator's health.

To achieve this, we added three labels to each of the operator alerts:

1. kubernetes_operator_part_of – operator name

2. kubernetes_operator_component – operator component (can be the same as the above)

3. operator_health_impact – indicates the impact of the issue on the operator's health. This label will get one of the following values:

• critical indicates an issue that negatively impacts the operator's health, and the user should take action immediately to solve it.
• warning indicates an issue that might soon negatively impact the operator's health, and the user should solve the issue.
• none indicates an issue that doesn't affect the operator's health. In most cases, this is related to the workload rather than the operator itself.

We also created and added a new metric that aggregates the metrics conditions by the name kubevirt_hco_system_health_status.

Its value is calculated by the following:

• Degraded = True  or !Available = True-> kubevirt_hco_system_health_status = 2 (Error)
• Progressing = True or !ReconcileComplete = True -> kubevirt_hco_system_health_status = 1 (Warning)
• Else (Available = True and Degraded = False and Progressing = False and ReconcileComplete = True) -> kubevirt_hco_system_health_status = 0 (Healthy)

We created a recording rule for the health metric called kubevirt_hyperconverged_operator_health_status that will indicate the overall health of our operator, link. Values: healthy (0), warning (1) or critical (2).
It is based on both alerts and operator conditions. Its value will be the minimum of the two mentioned above.

This metric provides a clear operator health status and enables you to easily detect the causes of the issue by checking the firing alerts and their health impact.

Health metrics based on Conditions vs. Alerts