Configure OpenShift Metrics with Prometheus backed by OpenShift Container Storage

Note: This scenario assumes you already have an OpenShift cluster or have followed the instructions in the Deploying OpenShift Container Storage 4 to OpenShift 4 Blog to set up an OpenShift Container Platform 4.2.14+ cluster using OpenShift Container Storage 4.

1. This Blog will cover:

What Prometheus is
Checking the current storage backend of your Prometheus environment
Make your monitoring and data persistent

2. What is Prometheus?

From the official Prometheus website:
“Prometheus was started in 2012 by Soundcloud and is an open-source monitoring and alerting toolkit. Nowadays it is a stand-alone project and independent of any single company. Due to its design, it doesn’t rely itself on a cluster or distributed database, but all its nodes are autonomous. All communication happens through HTTP and Prometheus pulls information from its’ nodes rather than receiving them like Nagios, for example.”

3. Modify your Prometheus environment

By default, Prometheus in Red Hat OpenShift Container Platform 4 is deployed on ephemeral storage so it is now time to talk about adjusting the environment to your needs. Every supported configuration change is controlled through a central ConfigMap, which needs to be created before we can make changes.

3.1. Create the ConfigMap

When you start off with a clean installation of Openshift, the ConfigMap to configure the Prometheus environment may not be present. To check if your ConfigMap is present, execute this:

oc -n openshift-monitoring get configmap cluster-monitoring-config

Output if the ConfigMap is not yet created:

Error from server (NotFound): configmaps "cluster-monitoring-config" not found

If you are missing the ConfigMap, create it:

oc -n openshift-monitoring create configmap cluster-monitoring-config

You can edit the ConfigMap with the following command. Do this now and ensure that the ConfigMap looks like below - especially the data section should be present:

oc -n openshift-monitoring edit configmap cluster-monitoring-config

ConfigMap content

apiVersion: v1

kind: ConfigMap

metadata:

  name: cluster-monitoring-config

  namespace: openshift-monitoring

data:

  config.yaml: |

3.2. Configuring persistent storage for the Prometheus stack

The Prometheus stack consists of the Prometheus database and the alertmanager data. Persisting the data from both is a best practice since data loss on any of these will cause you to lose your collected metrics and alerting data. View the official Openshift 4.2 documentation about this topic for further information.
While the documentation recommends using the local-storage provider, we will set up the Prometheus stack to use OpenShift Container Storage. By doing so, we will ensure that the Prometheus Pods can move freely between Nodes. Watch out for our performance briefs where we will show what this means for performance, by comparing the performance of the default EmptyDir, the recommended local-storage and OpenShift Container Storage-backed Prometheus.
To configure the Prometheus stack to use OpenShift Container Storage, edit the ConfigMap that was created in the previous section:

oc -n openshift-monitoring edit configmap cluster-monitoring-config

ConfigMap content

apiVersion: v1

kind: ConfigMap

metadata:

  name: cluster-monitoring-config

  namespace: openshift-monitoring

data:

  config.yaml: |

    prometheusK8s:

      volumeClaimTemplate:

        metadata:

          name: prometheusdb

        spec:

          storageClassName: ocs-storagecluster-ceph-rbd

          resources:

            requests:

              storage: 40Gi

    alertmanagerMain:

      volumeClaimTemplate:

        metadata:

          name: alertmanager

        spec:

          storageClassName: ocs-storagecluster-ceph-rbd

          resources:

            requests:

              storage: 40Gi

Once you save and exit the editor, the affected Pods will automatically be restarted and the new storage will be applied.

Note: It is not possible to retain data that was written on the default EmptyDir-based installation. Thus you will start with an empty database after changing the backend storage.
After a couple of minutes, the Alertmanager and Prometheus Pods will have restarted and you will see new PVCs in the openshift-monitoring namespace:

oc get -n openshift-monitoring pvc

Example output:

NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE

alertmanager-alertmanager-main-0   Bound    pvc-2f6714f7-feff-11e9-9bdd-005056818b15   40Gi       RWO            ocs-storagecluster-ceph-rbd   102m

alertmanager-alertmanager-main-1   Bound    pvc-2f6dd091-feff-11e9-9bdd-005056818b15   40Gi       RWO            ocs-storagecluster-ceph-rbd   102m

alertmanager-alertmanager-main-2   Bound    pvc-2f74e00d-feff-11e9-9bdd-005056818b15   40Gi       RWO            ocs-storagecluster-ceph-rbd   102m

prometheusdb-prometheus-k8s-0      Bound    pvc-e0f7b201-ff0c-11e9-9bdd-005056818b15   40Gi       RWO            ocs-storagecluster-ceph-rbd   4m34s

prometheusdb-prometheus-k8s-1      Bound    pvc-e101b1db-ff0c-11e9-9bdd-005056818b15   40Gi       RWO            ocs-storagecluster-ceph-rbd   4m34s

3.3. Configure even more

You can configure a lot more inside the cluster-monitoring-config ConfigMap. Since this Blog is focused on Storage, the other options have been omitted. A great way to learn more is to go to the official OpenShift Container Platform documentation for configuring the Prometheus Cluster Monitoring stack.
One thing you want to check out in the documentation is how you can set up the alertmanager.yml and how to define the retention time of Prometheus. By default, Prometheus only retains the last 15 days worth of data.

Resources and Feedback

To find out more about OpenShift Container Storage or to take a test drive, visit https://www.openshift.com/products/container-storage/.

If you would like to learn more about what the OpenShift Container Storage team is up to or provide feedback on any of the new 4.2 features, take this brief 3-minute survey.