How to Observe your Clusters with Red Hat Advanced Cluster Management - Customize the Grafana Dashboard

Introduction

With the Red Hat Advanced Cluster Management for Kubernetes (RHACM) observability service enabled, you can use RHACM to gain insight and optimize your managed clusters. This information can save cost and prevent unnecessary events.

When RHACM observability is installed, the metrics in managed clusters are collected and uploaded to the hub cluster. The default metrics that are collected are defined in the ConfigMap, observability-metrics-custom-allowlist. There are also predefined Grafana dashboards in RHACM, so that you can observe all the managed clusters from the hub cluster easily.

The previously mentioned default metrics collection and dashboards may not satisfy users. In this blog series, I introduce how to customize your RHACM observability settings. This article introduces how to customize Grafana dashboards. More specifically, I demonstrate how to customize the metrics allowlist and alert rules.

Prerequisites

• You must install Red Hat Advanced Cluster Management for Kubernetes (RHACM)
• You must enable the RHACM observability service

Default grafana dashboards

RHACM observability provides the ability to monitor the health of your managed clusters. You can observe and manage clusters through Grafana dashboards. Access the Grafana dashboards through the URL, https://$ACM_URL/grafana. The default Grafana dashboards are shown in the following image. See the source code in multicluster-observability-operator.

Customize the Grafana dashboards

RHACM observability also provides the ability to customize the Grafana dashboards. For example, the default dashboards do not show the average received container bandwidth, so you should add a dashboard and panel to observe that metric. Complete the following steps to design and create your own dashboard:

1. Clone the following repository, multicluster-observability-operator:

   git clone git@github.com:open-cluster-management/multicluster-observability-operator.git
   

2. Set up the Grafana developer instance:

   cd multicluster-observability-operator/tools
   ./setup-grafana-dev.sh --deploy
   

3. Switch the user to be a Grafana admin. Navigate to the Grafana URL, https://$ACM_URL/grafana-dev/ and log in. Then run the following command:

   ./switch-to-grafana-admin.sh kube:admin
   

4. Design your Grafana dashboard. For more information, see Design your Grafana dashboard. Complete the following steps to create a dashboard:

   • From the Grafana console, create a dashboard by selecting the Add icon from the navigation panel. Hover your cursor over Create > Dashboard. Then click Add new panel:

     ---

     

   • Select the dashboard settings. From the General tab, define the dashboard name as Custom Dashboard.

     ---

     

   • From the Variables tab, add the variable name datasource and cluster provider, which is used in the dashboard query. Fill in the table as seen in the following image:

     ---

     

     ---

     

   • From the Custom Dashboard/Edit Panel view, navigate to the Query tab.

   • Configure your query by selecting Observatorium from the data source selector and enter a PromQL query:

     avg(irate(container_network_receive_bytes_total{cluster="$cluster", namespace=~".+"}[$__interval])) by (namespace)
     

     ---

     

   • From the Grafana dashboard header, click the Save icon that is in the dashboard header.

   • Use the generate-dashboard-configmap-yaml.sh script to generate the dashboard ConfigMap, and to save the ConfigMap locally. Run the following command:

     # ./generate-dashboard-configmap-yaml.sh "Custom Dashboard"
     Save dashboard <custom-dashboard> to ./custom-dashboard.yaml
     

   For more information see, example/custom-dashboard.yaml for an existing custom dashboard.

So far, you can search and see a new dashboard named Custom Dashboard as shown in the following image:

Note: You may also see the panel has no data. This happens by default because the only metrics listed in observability-metrics-allowlist are collected by the RHACM observability service. To collect more metrics for your custom dashboard, continue reading or follow Adding custom metrics from the product documentation.

Update the custom metrics allowlist

In this example, let's update the custom metrics allowlist for the example/observability-metrics-custom-allowlist.yaml:

1. Add custom metrics with the following command:

   oc create -f example/observability-metrics-custom-allowlist.yaml
   

2. View and explore data by navigating to the Grafana explore page, and search the collected metrics data. Your results might resemble the following image:

   ---

   

3. Let's go back to the Custom Dashboard dashboard to see that the data is available in the panel:

   ---

   

Summary

Throughout this blog, I have provided guidance to help you design your Grafana dashboard through the console. Remember to update the metrics allowlist if your dashboard is using a new metrics name. In the next article, I plan to introduce more on how to customize the metrics collection.