Note: This scenario assumes you already have an OpenShift cluster or have followed the instructions in the Deploying OpenShift Container Storage 4 to OpenShift 4 Blog to set up an OpenShift Container Platform 4.2.14+ cluster using OpenShift Container Storage 4.
1. This Blog will cover:
What Prometheus is
Checking the current storage backend of your Prometheus environment
Make your monitoring and data persistent
2. What is Prometheus?
From the official Prometheus website:
“Prometheus was started in 2012 by Soundcloud and is an open-source monitoring and alerting toolkit. Nowadays it is a stand-alone project and independent of any single company. Due to its design, it doesn’t rely itself on a cluster or distributed database, but all its nodes are autonomous. All communication happens through HTTP and Prometheus pulls information from its’ nodes rather than receiving them like Nagios, for example.”
3. Modify your Prometheus environment
By default, Prometheus in Red Hat OpenShift Container Platform 4 is deployed on ephemeral storage so it is now time to talk about adjusting the environment to your needs. Every supported configuration change is controlled through a central ConfigMap, which needs to be created before we can make changes.
3.1. Create the ConfigMap
When you start off with a clean installation of Openshift, the ConfigMap to configure the Prometheus environment may not be present. To check if your ConfigMap is present, execute this:
oc -n openshift-monitoring get configmap cluster-monitoring-config
Output if the ConfigMap is not yet created:
Error from server (NotFound): configmaps "cluster-monitoring-config" not found
If you are missing the ConfigMap, create it:
oc -n openshift-monitoring create configmap cluster-monitoring-config
You can edit the ConfigMap with the following command. Do this now and ensure that the ConfigMap looks like below - especially the data section should be present:
oc -n openshift-monitoring edit configmap cluster-monitoring-config
ConfigMap content
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
3.2. Configuring persistent storage for the Prometheus stack
The Prometheus stack consists of the Prometheus database and the alertmanager data. Persisting the data from both is a best practice since data loss on any of these will cause you to lose your collected metrics and alerting data. View the official Openshift 4.2 documentation about this topic for further information.
While the documentation recommends using the local-storage provider, we will set up the Prometheus stack to use OpenShift Container Storage. By doing so, we will ensure that the Prometheus Pods can move freely between Nodes. Watch out for our performance briefs where we will show what this means for performance, by comparing the performance of the default EmptyDir, the recommended local-storage and OpenShift Container Storage-backed Prometheus.
To configure the Prometheus stack to use OpenShift Container Storage, edit the ConfigMap that was created in the previous section:
oc -n openshift-monitoring edit configmap cluster-monitoring-config
ConfigMap content
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
prometheusK8s:
volumeClaimTemplate:
metadata:
name: prometheusdb
spec:
storageClassName: ocs-storagecluster-ceph-rbd
resources:
requests:
storage: 40Gi
alertmanagerMain:
volumeClaimTemplate:
metadata:
name: alertmanager
spec:
storageClassName: ocs-storagecluster-ceph-rbd
resources:
requests:
storage: 40Gi
Once you save and exit the editor, the affected Pods will automatically be restarted and the new storage will be applied.
Note: It is not possible to retain data that was written on the default EmptyDir-based installation. Thus you will start with an empty database after changing the backend storage.
After a couple of minutes, the Alertmanager and Prometheus Pods will have restarted and you will see new PVCs in theopenshift-monitoringnamespace:
oc get -n openshift-monitoring pvc
Example output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
alertmanager-alertmanager-main-0 Bound pvc-2f6714f7-feff-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 102m
alertmanager-alertmanager-main-1 Bound pvc-2f6dd091-feff-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 102m
alertmanager-alertmanager-main-2 Bound pvc-2f74e00d-feff-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 102m
prometheusdb-prometheus-k8s-0 Bound pvc-e0f7b201-ff0c-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 4m34s
prometheusdb-prometheus-k8s-1 Bound pvc-e101b1db-ff0c-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 4m34s
3.3. Configure even more
You can configure a lot more inside the cluster-monitoring-config ConfigMap. Since this Blog is focused on Storage, the other options have been omitted. A great way to learn more is to go to the official OpenShift Container Platform documentation for configuring the Prometheus Cluster Monitoring stack.
One thing you want to check out in the documentation is how you can set up the alertmanager.yml and how to define the retention time of Prometheus. By default, Prometheus only retains the last 15 days worth of data.
Resources and Feedback
To find out more about OpenShift Container Storage or to take a test drive, visit https://www.openshift.com/products/container-storage/.
If you would like to learn more about what the OpenShift Container Storage team is up to or provide feedback on any of the new 4.2 features, take this brief 3-minute survey.
关于作者
Red Hatter since 2018, technology historian and founder of The Museum of Art and Digital Entertainment. Two decades of journalism mixed with technology expertise, storytelling and oodles of computing experience from inception to ewaste recycling. I have taught or had my work used in classes at USF, SFSU, AAU, UC Law Hastings and Harvard Law.
I have worked with the EFF, Stanford, MIT, and Archive.org to brief the US Copyright Office and change US copyright law. We won multiple exemptions to the DMCA, accepted and implemented by the Librarian of Congress. My writings have appeared in Wired, Bloomberg, Make Magazine, SD Times, The Austin American Statesman, The Atlanta Journal Constitution and many other outlets.
I have been written about by the Wall Street Journal, The Washington Post, Wired and The Atlantic. I have been called "The Gertrude Stein of Video Games," an honor I accept, as I live less than a mile from her childhood home in Oakland, CA. I was project lead on the first successful institutional preservation and rebooting of the first massively multiplayer game, Habitat, for the C64, from 1986: https://neohabitat.org . I've consulted and collaborated with the NY MOMA, the Oakland Museum of California, Cisco, Semtech, Twilio, Game Developers Conference, NGNX, the Anti-Defamation League, the Library of Congress and the Oakland Public Library System on projects, contracts, and exhibitions.
