Subscribe to the feed

Prometheus is one of the fastest Cloud Native Computing Foundation projects being adopted. And now it comes as a native product in the OpenShift stack. It gives you everything that a good enterprise monitoring tool needs in one place: Good API, easy integration, time series database, real-time data, alerting, and flexibility. And having good visibility into your cluster state is one of the most important things about operating a distributed system.

In this guide, we will configure OpenShift Prometheus to send email alerts. In addition, we will configure Grafana dashboard to show some basic metrics. All components—Prometheus, NodeExporter, and Grafana—will be created in the separate projects. Prometheus web UI and AlertManager UI will be used only for configuration and testing. Our OpenShift cluster already has Prometheus deployed using Ansible Playbooks.
Ansible Playbook will create 1 pod with 5 containers running. This will change in the future when the HA solution for Prometheus and AlertManager is developed. But for now, this will do :)

We will deploy node-exporter so we can have some node level metrics too.

oc adm new-project openshift-metrics-node-exporter --node-selector='zone=az1'
oc project openshift-metrics-node-exporter
oc create -f -n openshift-metrics-node-exporter
oc adm policy add-scc-to-user -z prometheus-node-exporter -n openshift-metrics-node-exporter hostaccess

To deploy Grafana we will use an already existing project from mrsiano github project.

git clone
cd grafana-ocp
./ prometheus-ocp openshift-metrics true

To configure Grafana to consume Prometheus, we will link grafana and openshift-metrics projects:

oc adm pod-network join-projects --to=grafana openshift-metrics

For the authentication read management-admin service account token:

oc sa get-token management-admin -n management-infra

Login to the Grafana dashboard and add new source:


Example queries were taken from OpensShift Git repository.

  • For cluster memory: sum(container_memory_rss) / sum(machine_memory_bytes)
  • For changes in the cluster: sum(changes(container_start_time_seconds[10m]))
  • For API calls: sum(apiserver_request_count{verb=~"POST|PUT|DELETE|PATCH"}[5m]) and sum(apiserver_request_count{verb=~"GET|LIST|WATCH"}[5m])

Next, we will configure the alerts and mail client. For mail messages, I will use as it gives an easy way to send mail via SMTP.

Now the alert manager is not exposed to outside world, so we will expose it temporarily in order to see alerts being generated.

cat <<EOF | oc create -n openshift-metrics -f -
apiVersion: v1
kind: Service
name: alertmanager
name: alertmanager
namespace: openshift-metrics
- port: 8080
protocol: TCP
targetPort: 9093
app: prometheus
sessionAffinity: None
type: ClusterIP
loadBalancer: {}

oc expose service alertmanager -n openshift-metrics
route="$(oc get routes -n openshift-metrics alertmanager --template='{{ }}')"
echo "Try accessing $route"

We need to modify Prometheus configMap to enable our new rules for alerts. We execute oc edit cm prometheus and add this:

prometheus.rules: |
- name: example-rules
interval: 30s # defaults to global interval
- alert: "Node Down"
expr: up{job="kubernetes-nodes"} == 0
miqTarget: "ContainerNode"
severity: "ERROR"
url: ""
message: "Node {{$labels.instance}} is down"
- alert: "Too Many Pods"
expr: sum(kubelet_running_pod_count) > 15
miqTarget: "ExtManagementSystem"
severity: "ERROR"
url: ""
message: "Too many running pods"
- alert: "Node CPU Usage"
expr: (100 - (avg by (instance) (irate(node_cpu{app="prometheus-node-exporter",mode="idle"}[5m])) * 100)) > 3
for: 30s
severity: "ERROR"
miqTarget: "ExtManagementSystem"
severity: "ERROR"
url: ""
message: "{{$labels.instance}}: CPU usage is above 4% (current value is: {{ $value }})"

This will create an alert on running pods count, CPU usage, and if the node is down. We can check the same data in the Prometheus web UI:(100 - (avg by (instance) (irate(node_cpu{app="prometheus-node-exporter",mode="idle"}[5m])) * 100)) > 4 and sum(kubelet_running_pod_count)

Now we will configure our mail client for the alert manager. Edit configMap oc edit cm prometheus-alerts and modify it to look like this:

apiVersion: v1
alertmanager.yml: |
smtp_smarthost: ''
smtp_from: ''
smtp_auth_username: 'xxxxx'
smtp_auth_password: 'xxxxx'
# The root route on which each incoming alert enters.
# default route if none match
receiver: all
# The labels by which incoming alerts are grouped together. For example,
# multiple alerts coming in for cluster=A and alertname=LatencyHigh would
# be batched into a single group.
group_by: []
# All the above attributes are inherited by all child routes and can
# overwritten on each.
- name: alert-buffer-wh
- url: http://localhost:9099/topics/alerts
- name: mail
- to: ""
- name: all
- to: ""
- url: http://localhost:9099/topics/alerts
kind: ConfigMap
creationTimestamp: null
name: prometheus-alerts

This will send alert to both backends so they will be available via API of the alert-buffer and mail. The alert buffer API shows alerts being triggered:

Prometheus alerts

And after few seconds emails start coming through:
Prometheus emails
It is an early stage for Prometheus in OpenShift. But even with the current configuration, the value it brings to the platform is huge. Having real-time monitoring with close to real-time alerts is the key to the success when operating a distributed system. If you aren't convinced yet of the importance of real-time alerting, more arguments can be found in the Alerting from Time-Series Data blog post from our friends at Google.

About the author


Browse by channel

automation icon


The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon


The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon


The latest on the world’s leading enterprise Linux platform

application development icon


Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech