How to use the Red Hat OpenShift Data Foundation in Red Hat Advanced Cluster Management for Kubernetes Observability Service

Red Hat Advanced Cluster Management for Kubernetes (RHACM) observability service provides many ways to help you gain insight about your managed clusters, and optimize your managed clusters. This information can save costs and prevent unnecessary events. By default, observability service is included with the product installation, but not enabled. Due to the requirement for persistent storage, the observability service is not enabled by default. In this blog, I introduce you to how to use Red Hat OpenShift Data Foundation as stable object stores to store your metric data.

Red Hat OpenShift Data Foundation is software-defined storage for containers. Engineered as the data and storage services platform for Red Hat OpenShift, Red Hat OpenShift Data Foundation helps teams develop and deploy applications quickly and efficiently across clouds.

In this blog, I provide the steps for you to complete these high-level steps to configure your RHACM observabilty service with Red Hat OpenShift Data Foundation:

1. Deploy Red Hat OpenShift OpenShift Data Foundation.
2. Enable Red Hat Advanced Cluster Management observabilty service with Red Hat OpenShift Data Foundation.

Prerequisites

You need the following things configured in your environment:

• Configure oc (ver. 4.3+) and kubectl (ver. 1.16+) to connect to your Red Hat OpenShift Container Platform (Note: this name has changed to OpenShift Data Foundation in version 4) cluster
• Install RHACM
• Install jq

Deploy Red Hat OpenShift Data Foundation

Follow this blog, Deploying your storage backend using OpenShift Container Storage 4 to deploy Red Hat OpenShift Data Foundation.

Enable Red Hat Advanced Cluster Management Observabilty Service with Red Hat OpenShift Data Foundation

After the OpenShift Data Foundation is installed, complete the following steps to enable the RHACM observabilty service with Red Hat OpenShift Data Foundation:

1. Log in to your RHACM hub cluster.

2. Create a namespace for the observability service with the following command:

   oc create namespace open-cluster-management-observability
   

3. Generate your pull-secret. If RHACM is installed in the open-cluster-management namespace, run the following command:

   DOCKER_CONFIG_JSON=`oc extract secret/multiclusterhub-operator-pull-secret -n open-cluster-management --to=-`
   

   If the multiclusterhub-operator-pull-secret is not defined in the namespace, copy the pull-secret from the openshift-config namespace into the open-cluster-management-observability namespace. Run the following command:

   DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-`
   

   Then, create the pull-secret in the open-cluster-management-observability namespace, run the following command:

   oc create secret generic multiclusterhub-operator-pull-secret \
       -n open-cluster-management-observability \
       --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
       --type=kubernetes.io/dockerconfigjson
   

4. Access the object storage configuration from your OpenShift Data Foundation service, and then create the secret for your object storage:

   • Run the following command to get access key:

     $ ACCESS_KEY=$(oc get secret noobaa-admin -n openshift-storage -o json | jq -r '.data.AWS_ACCESS_KEY_ID|@base64d')
     
     $ echo $ACCESS_KEY
     YOUR_ACCESS_KEY_VALUE
     

   • Run the following command to get secret access key:

     $ SECRET_KEY=$(oc get secret noobaa-admin -n openshift-storage -o json | jq -r '.data.AWS_SECRET_ACCESS_KEY|@base64d')
     
     $ echo $SECRET_KEY
     YOUR_SECRET_KEY_VALUE
     

   • Run the following command to get endpoint:

     oc get noobaa -n openshift-storage -o yaml
     
     ...
     
        serviceS3:
            externalDNS:
            - YOUR_END_POINT_URL_VALUE
            - YOUR_END_POINT_URL_VALUE
     ...
     

     Your secret might resemble the following file:

     apiVersion: v1
     kind: Secret
     metadata:
       name: thanos-object-storage
       namespace: open-cluster-management-observability
     type: Opaque
     stringData:
       thanos.yaml: |
         type: s3
         config:
           bucket: first.bucket
           endpoint: YOUR_END_POINT_URL_VALUE
           insecure: true
           access_key: YOUR_ACCESS_KEY_VALUE
           secret_key: YOUR_SECRET_KEY_VALUE
     

   • Run the following command to create the secret for your object storage:

     oc apply -f thanos-object-storage.yaml -n open-cluster-management-observability
     

5. Create the MultiClusterObservability custom resource to enable the observability service. Your custom resource might resemble the following file:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
     annotations:
   spec:
     observabilityAddonSpec:
       enableMetrics: true
       interval: 30
     storageConfig:
       metricObjectStorage:
         key: thanos.yaml
         name: thanos-object-storage
   

   • Run the following command to create your custom resource:

     oc apply -f multiclusterobservability_cr.yaml
     

   • Check the custom resource status. If you receive the following message, the observability service is enabled successfully: Observability components are deployed and running. Run the following command to view the status:

     oc get mco observability -o yaml
     
     ...
     status:
       conditions:
       - lastTransitionTime: "2021-06-10T07:57:23Z"
         message: Installation is in progress
         reason: Installing
         status: "True"
         type: Installing
       - lastTransitionTime: "2021-06-10T07:59:24Z"
         message: Observability components are deployed and running
         reason: Ready
         status: "True"
         type: Ready
     

Conclusion

Throughout the blog, I introduce a storage solution for Red Hat Advanced Cluster Management observabilty service. In addition, you can also use Amazon S3 (or other S3 compatible object stores like Ceph), Google Cloud Storage, or Azure storage as the storage service. For more details on Red Hat Advanced Cluster Management observabilty service, see the product documentation.

I hope that this article was useful to you. Please don’t hesitate to reach out if you have any feedback!