If you're a Site Reliability Engineer, or do similar work, you've probably heard about eBPF. You might also have tried out a few bcc or bpftrace tools. Have you wondered how you can run these tools 24/7, log historical data and set alerts based on the measured metric values? This post will guide you through setting up Performance Co-Pilot, our monitoring solution for RHEL, and enabling eBPF sourced metrics on RHEL 8.
What is Performance Co-Pilot?
Performance Co-Pilot (PCP) is a system performance monitoring toolkit. It follows the Unix design philosophy of multiple small, focussed components that work together:
pmcd: The Performance Metrics Collection Daemon collects performance metrics from different PMDAs
pmda: Each Performance Metrics Domain Agent gathers metrics from a specific domain, for example from the /proc filesystem, from the MySQL database server or from bcc tools
pmie: The Performance Metrics Inference Engine connects to pmcd and performs actions based on rules (for example "run this shell script if the load average of the last 5 minutes was above 2")
pmlogger: connects to pmcd and creates archive logs of metric values
pmproxy: acts as a protocol proxy, provides a REST API and exports all metrics in the OpenMetrics format (in other words, you can scrape PCP metrics with Prometheus or OpenShift Monitoring)
For this post, we’ll focus on the different PMDAs. After all, we want to ingest eBPF metrics into our monitoring tool.
Let’s get started by installing PCP and reading our first metric:
$ sudo dnf install -y pcp-zeroconf $ pmval disk.dev.write_bytes metric: disk.dev.write_bytes host: agerstmayr-thinkpad semantics: cumulative counter (converting to rate) units: Kbyte (converting to Kbyte / sec) samples: all nvme0n1 sda 0.0 0.0 6219. 0.0 0.0 0.0 0.0 0.0 47.94 0.0 0.0 0.0
This output shows the amount of bytes written to each disk in KB/s in close to real time. This command reports the metric values every second, and can be stopped by pressing Ctrl + C. The same command can be used to access the metric values of a different machine or to access historical values of an archive created by pmlogger. Furthermore, the reporting interval can be adjusted and much more.
How to source metrics from eBPF?
There are multiple ways to write eBPF programs. The most popular eBPF front ends for monitoring programs are currently bcc (eBPF compiler collection), bpftrace and libbpf. PCP includes an agent for each front end, so you can use any of these front ends to gather metrics from eBPF programs.
bcc tools are written in Python (or Lua) with embedded eBPF/C code. The embedded eBPF/C code gets compiled at runtime, therefore it is required to have the Clang/LLVM compiler installed on the target machine. Additionally, kernel headers or the kheaders module need to be present on the target machine. The bcc PMDA is available since RHEL 7.6.
Let’s install the bcc PMDA and read our first eBPF sourced metrics:
$ sudo dnf install -y pcp-pmda-bcc $ cd /var/lib/pcp/pmdas/bcc && sudo ./Install $ pminfo bcc bcc.proc.net.udp.recv.bytes bcc.proc.net.udp.recv.calls bcc.proc.net.udp.send.bytes bcc.proc.net.udp.send.calls bcc.proc.net.tcp.recv.bytes bcc.proc.net.tcp.recv.calls bcc.proc.net.tcp.send.bytes bcc.proc.net.tcp.send.calls
The pminfo command lists all currently available metrics. In this case, the bcc PMDA already started one bcc tool, which measures per-process network bandwidth. Let’s install more bcc tools: open
/var/lib/pcp/pmdas/bcc/bcc.conf in your favorite editor and edit the modules option of the
[pmda] modules = netproc,runqlat,biolatency,tcptop,tcplife
Most bcc PMDA modules are based on the upstream bcc tools with small modifications (i.e. instead of command line arguments, each module can be configured in the
bcc.conf file). All supported bcc modules are installed in
/var/lib/pcp/pmdas/bcc/modules (there are 28 modules available in the latest version of PCP).
After updating the configuration file, reinstall the bcc PMDA by executing
cd /var/lib/pcp/pmdas/bcc && sudo ./Install, wait a few seconds until all bcc modules are compiled and run
pminfo bcc again. You’ll see a list of 27 different metrics. You can inspect them with pmval, pminfo or visually with Grafana.Figure 1: runqlat and biolatency tools visualized in GrafanaFigure 2: tcptop and tcplife tools visualized in Grafana
For instructions on how to setup Grafana, please refer to this blog post: Visualizing System Performance or consult the RHEL documentation at Setting up graphical representation of PCP metrics.
bpftrace is a high-level tracing language with a syntax influenced by awk and DTrace. It requires Clang/LLVM and kernel type information (the kernel-devel package, kheaders module or BTF) available on the target machine. The bpftrace PMDA is available since RHEL 8.2.
This PMDA supports two modes of operation: It starts all bpftrace scripts located in the
/var/lib/pcp/pmdas/bpftrace/autostart directory and supports starting bpftrace scripts on-demand in Grafana with the bpftrace data source. The second mode of operation is intended for development/test purposes only, and not recommended in production environments (and disabled by default).
The bpftrace PMDA starts each bpftrace script and exports all bpftrace variables of the script and additional status information about each bpftrace process as PCP metrics:
$ sudo dnf install -y pcp-pmda-bpftrace $ cd /var/lib/pcp/pmdas/bpftrace && sudo ./Install $ pminfo bpftrace bpftrace.scripts.runqlat.data.usecs bpftrace.scripts.runqlat.data_bytes bpftrace.scripts.runqlat.code bpftrace.scripts.runqlat.probes bpftrace.scripts.runqlat.error bpftrace.scripts.runqlat.exit_code bpftrace.scripts.runqlat.pid bpftrace.scripts.runqlat.status bpftrace.scripts.biolatency.data.usecs bpftrace.scripts.biolatency.data_bytes bpftrace.scripts.biolatency.code bpftrace.scripts.biolatency.probes bpftrace.scripts.biolatency.error bpftrace.scripts.biolatency.exit_code bpftrace.scripts.biolatency.pid bpftrace.scripts.biolatency.status $ pminfo -dfT bpftrace.scripts.biolatency.data.usecs bpftrace.scripts.biolatency.data.usecs Data Type: 64-bit unsigned int InDom: 151.100010 0x25c186aa Semantics: counter Units: none Help: @usecs variable of bpftrace script inst [0 or "8-15"] value 1504 inst [1 or "16-31"] value 58274 inst [2 or "32-63"] value 64363 inst [3 or "64-127"] value 220222 inst [4 or "128-255"] value 87631 inst [5 or "256-511"] value 31494 inst [6 or "512-1023"] value 5101
Outlook: bpf PMDA
Another way of writing eBPF programs is using BPF CO-RE (Compile Once, Run Everywhere). BPF CO-RE requires BTF (eBPF Type Format), libbpf and a recent kernel. By combining these technologies, eBPF applications can be compiled once and be executed on target machines without any compiler or kernel headers installed.
The programs are written in C, making them a lightweight alternative to the bcc based tools. This PMDA is planned to be available in RHEL 9.
This post gave a quick overview of the current eBPF monitoring front ends and how to integrate metrics from them into PCP. By integrating eBPF metrics in the PCP ecosystem, Site Reliability Engineers can quickly access eBPF metrics from remote hosts, can store historical metric values to disk (pmlogger), set alerts based on metric values (pmie) and visualize metrics in Grafana (grafana-pcp).
Further Reading: For detailed instructions on how to set up Grafana with Performance Co-Pilot, please refer to Visualizing System Performance. For general performance related blog posts, follow our Performance Optimization Series.
Thanks to Marko Myllynen (bcc PMDA), Jason Koch (bpf PMDA), Brendan Gregg (bcc and bpftrace tools) and the PCP, eBPF and Linux communities.