The Ceph monitoring challenge: Prometheus, Grafana, and Ansible rise to the task

Managing growth in applications and infrastructure is a significant challenge for many organizations as they scale. Every new application or server that’s added to the environment serves a business requirement, so keeping services usable in today’s 24x7x365 world represents both a challenge and an opportunity to meet a higher service level agreement (SLA).

In part 1 of this series, we discussed a Software-as-a-Solution (SaaS) solution to monitor your Ceph storage infrastructure. In this post, we continue the discussion by talking about how Prometheus, Grafana, and Ansible can rise to the challenge of Ceph monitoring.

Understanding the Ceph monitoring challenge

In the past, monitoring a service typically resulted in separate platforms. In today’s enterprise architectures, however, this is both undesirable and operationally impractical. Today’s monitoring fabrics must span legacy architectures as well as modern applications.

Monitoring software-defined storage (SDS) platforms like Red Hat Ceph Storage must fit the context of today’s enterprise monitoring architectures. Red Hat Ceph Storage is an infrastructure layer that may be called on to support hundreds or thousands of applications and users, so monitoring should  be simple to deploy and offer advanced correlation and alerting capabilities.

Let’s look at how the tools integrated in Red Hat Ceph Storage can rise to the monitoring challenge.

Laying out our Ceph monitoring strategy

Effective telemetry starts with accurate, timely, and reliable data. Sourcing this data as efficiently as possible is a key operational goal. In the past, agents have been used to gather metrics and forward them to a monitoring engine.

We can go one better with Red Hat Ceph Storage 3, by instrumenting the storage layer itself, with the help of the Prometheus project.

Delving into data collection

Prometheus implements a two-tier architecture: server(s) and exporters. The exporters provide metrics to the server and use standard HTTP. They can be implemented as either standalone daemons or embedded within the application layer itself. With Red Hat Ceph Storage, metrics are provided by an embedded exporter module, which means there are:

• No agents to deploy

• No agents to manage (start/stop/debug)

• Available metrics that track the current Red Hat Ceph Storage release (reducing the lag between the monitoring stack and the available metrics)

• More seamless support for container and RPM-based deployments

One of the other key benefits of the embedded exporter architecture is that data gathering inherits higher availability from the Ceph Manager daemon.

Telling the cluster to provide monitoring telemetry can be simple: Just enable the Prometheus manager module:

> ceph mgr module enable prometheus

Once that is done, the prometheus server can be configured to “scrape” data from Red Hat Ceph Storage. Following is a sample scrape definition (prometheus.yml)that shows how you define each of the ceph mgr daemons as a data source, but link them through a common instance name (‘ceph_cluster’).

scrape_configs:  
  - job_name: 'ceph'
    honor_labels: true
    static_configs:
      - targets: ['rhceph-1:9283']
        labels:
          instance: 'ceph_cluster'
      - targets: ['rhceph-3:9283']
        labels:
          instance: 'ceph_cluster'
      - targets: ['rhceph-2:9283']
        labels:
          instance: 'ceph_cluster'

Examples like these help improve our understanding of how things work, but when it comes to actually deploying monitoring in Red Hat Ceph Storage, we would use Ansible to automate the legwork.

Metrics from Ceph only really represent part of the picture. With Red Hat Ceph Storage, the deployment process also installs the prometheus node_exporter daemon on the Red Hat Ceph Storage hosts. With the node_exporter in place, monitoring of the storage cluster is enhanced to include insights from OS-level metrics, like CPU, RAM, network, and disk.

With this level of integration, operations teams can deep-dive into storage issues and move between storage metrics and OS-level metrics to diagnose unexpected behavior or even predict performance.

Getting into visualization

Having the data available is a fundamental requirement, but being able to graph the data in a meaningful way is just as important. In addition to the Prometheus server, a Red Hat Ceph Storage cluster includes an instance of Grafana, an open source analytics and visualization tool. The Grafana instance provides several dashboards for visualization, a central dashboard for alert triggers and a separate dashboard focused on the performance of the monitoring host itself.

Each dashboard has been designed to provide insights into Ceph cluster operations, enabling the user to answer questions like:

• How effective is the external load-balancer distributing S3 connections to my cluster?

• How busy is my workload making the physical drives in the servers?

• What kind of I/O profile is a storage pool dealing with (R:W ratio)?

• Which is my busiest pool or heaviest workload?

The next diagram shows how the dashboards relate to each other.