Note: This scenario assumes you already have an OpenShift cluster or have followed the instructions in the Deploying OpenShift Container Storage 4 to OpenShift 4 Blog to set up an OpenShift Container Platform 4.2.14+ cluster using OpenShift Container Storage 4.
1. This Blog will cover:
What Prometheus is
Checking the current storage backend of your Prometheus environment
Make your monitoring and data persistent
2. What is Prometheus?
From the official Prometheus website:
“Prometheus was started in 2012 by Soundcloud and is an open-source monitoring and alerting toolkit. Nowadays it is a stand-alone project and independent of any single company. Due to its design, it doesn’t rely itself on a cluster or distributed database, but all its nodes are autonomous. All communication happens through HTTP and Prometheus pulls information from its’ nodes rather than receiving them like Nagios, for example.”
3. Modify your Prometheus environment
By default, Prometheus in Red Hat OpenShift Container Platform 4 is deployed on ephemeral storage so it is now time to talk about adjusting the environment to your needs. Every supported configuration change is controlled through a central ConfigMap
, which needs to be created before we can make changes.
3.1. Create the ConfigMap
When you start off with a clean installation of Openshift, the ConfigMap
to configure the Prometheus environment may not be present. To check if your ConfigMap
is present, execute this:
oc -n openshift-monitoring get configmap cluster-monitoring-config
Output if the ConfigMap
is not yet created:
Error from server (NotFound): configmaps "cluster-monitoring-config" not found
If you are missing the ConfigMap
, create it:
oc -n openshift-monitoring create configmap cluster-monitoring-config
You can edit the ConfigMap with the following command. Do this now and ensure that the ConfigMap looks like below - especially the data section should be present:
oc -n openshift-monitoring edit configmap cluster-monitoring-config
ConfigMap content
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
3.2. Configuring persistent storage for the Prometheus stack
The Prometheus stack consists of the Prometheus database and the alertmanager data. Persisting the data from both is a best practice since data loss on any of these will cause you to lose your collected metrics and alerting data. View the official Openshift 4.2 documentation about this topic for further information.
While the documentation recommends using the local-storage provider, we will set up the Prometheus stack to use OpenShift Container Storage. By doing so, we will ensure that the Prometheus Pods can move freely between Nodes. Watch out for our performance briefs where we will show what this means for performance, by comparing the performance of the default EmptyDir, the recommended local-storage and OpenShift Container Storage-backed Prometheus.
To configure the Prometheus stack to use OpenShift Container Storage, edit the ConfigMap that was created in the previous section:
oc -n openshift-monitoring edit configmap cluster-monitoring-config
ConfigMap content
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
prometheusK8s:
volumeClaimTemplate:
metadata:
name: prometheusdb
spec:
storageClassName: ocs-storagecluster-ceph-rbd
resources:
requests:
storage: 40Gi
alertmanagerMain:
volumeClaimTemplate:
metadata:
name: alertmanager
spec:
storageClassName: ocs-storagecluster-ceph-rbd
resources:
requests:
storage: 40Gi
Once you save and exit the editor, the affected Pods will automatically be restarted and the new storage will be applied.
Note: It is not possible to retain data that was written on the default EmptyDir-based installation. Thus you will start with an empty database after changing the backend storage.
After a couple of minutes, the Alertmanager and Prometheus Pods will have restarted and you will see new PVCs in theopenshift-monitoring
namespace:
oc get -n openshift-monitoring pvc
Example output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
alertmanager-alertmanager-main-0 Bound pvc-2f6714f7-feff-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 102m
alertmanager-alertmanager-main-1 Bound pvc-2f6dd091-feff-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 102m
alertmanager-alertmanager-main-2 Bound pvc-2f74e00d-feff-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 102m
prometheusdb-prometheus-k8s-0 Bound pvc-e0f7b201-ff0c-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 4m34s
prometheusdb-prometheus-k8s-1 Bound pvc-e101b1db-ff0c-11e9-9bdd-005056818b15 40Gi RWO ocs-storagecluster-ceph-rbd 4m34s
3.3. Configure even more
You can configure a lot more inside the cluster-monitoring-config
ConfigMap. Since this Blog is focused on Storage, the other options have been omitted. A great way to learn more is to go to the official OpenShift Container Platform documentation for configuring the Prometheus Cluster Monitoring stack.
One thing you want to check out in the documentation is how you can set up the alertmanager.yml
and how to define the retention time of Prometheus. By default, Prometheus only retains the last 15 days worth of data.
Resources and Feedback
To find out more about OpenShift Container Storage or to take a test drive, visit https://www.openshift.com/products/container-storage/.
If you would like to learn more about what the OpenShift Container Storage team is up to or provide feedback on any of the new 4.2 features, take this brief 3-minute survey.
À propos de l'auteur
Red Hatter since 2018, technology historian and founder of The Museum of Art and Digital Entertainment. Two decades of journalism mixed with technology expertise, storytelling and oodles of computing experience from inception to ewaste recycling. I have taught or had my work used in classes at USF, SFSU, AAU, UC Law Hastings and Harvard Law.
I have worked with the EFF, Stanford, MIT, and Archive.org to brief the US Copyright Office and change US copyright law. We won multiple exemptions to the DMCA, accepted and implemented by the Librarian of Congress. My writings have appeared in Wired, Bloomberg, Make Magazine, SD Times, The Austin American Statesman, The Atlanta Journal Constitution and many other outlets.
I have been written about by the Wall Street Journal, The Washington Post, Wired and The Atlantic. I have been called "The Gertrude Stein of Video Games," an honor I accept, as I live less than a mile from her childhood home in Oakland, CA. I was project lead on the first successful institutional preservation and rebooting of the first massively multiplayer game, Habitat, for the C64, from 1986: https://neohabitat.org . I've consulted and collaborated with the NY MOMA, the Oakland Museum of California, Cisco, Semtech, Twilio, Game Developers Conference, NGNX, the Anti-Defamation League, the Library of Congress and the Oakland Public Library System on projects, contracts, and exhibitions.
Parcourir par canal
Automatisation
Les dernières nouveautés en matière d'automatisation informatique pour les technologies, les équipes et les environnements
Intelligence artificielle
Actualité sur les plateformes qui permettent aux clients d'exécuter des charges de travail d'IA sur tout type d'environnement
Cloud hybride ouvert
Découvrez comment créer un avenir flexible grâce au cloud hybride
Sécurité
Les dernières actualités sur la façon dont nous réduisons les risques dans tous les environnements et technologies
Edge computing
Actualité sur les plateformes qui simplifient les opérations en périphérie
Infrastructure
Les dernières nouveautés sur la plateforme Linux d'entreprise leader au monde
Applications
À l’intérieur de nos solutions aux défis d’application les plus difficiles
Programmes originaux
Histoires passionnantes de créateurs et de leaders de technologies d'entreprise
Produits
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Services cloud
- Voir tous les produits
Outils
- Formation et certification
- Mon compte
- Assistance client
- Ressources développeurs
- Rechercher un partenaire
- Red Hat Ecosystem Catalog
- Calculateur de valeur Red Hat
- Documentation
Essayer, acheter et vendre
Communication
- Contacter le service commercial
- Contactez notre service clientèle
- Contacter le service de formation
- Réseaux sociaux
À propos de Red Hat
Premier éditeur mondial de solutions Open Source pour les entreprises, nous fournissons des technologies Linux, cloud, de conteneurs et Kubernetes. Nous proposons des solutions stables qui aident les entreprises à jongler avec les divers environnements et plateformes, du cœur du datacenter à la périphérie du réseau.
Sélectionner une langue
Red Hat legal and privacy links
- À propos de Red Hat
- Carrières
- Événements
- Bureaux
- Contacter Red Hat
- Lire le blog Red Hat
- Diversité, équité et inclusion
- Cool Stuff Store
- Red Hat Summit