Understanding the performance of your own applications with RHEL: it’s easy with PCP

With Red Hat Enterprise Linux (RHEL) 8.2, we have extended the tools for visualizing performance data--so called “metrics”. These metrics can be collected by PCP, the Performance Co-Pilot framework. PCP comes with code to collect metrics from a wide area of commonly used software: Postgresql databases, systems temperature sensors, KVM guest stats, Apache httpd and so on; all of these can be monitored out of the box. These code pieces specialized in an area of monitoring are Performance Metric Domain Agents (PMDA), with package names pmda-apache, pmda-postgresql and so on.

But what if the RHEL repositories have no PMDA for an application? Our customers run all kinds of applications, so how to get an overview of their performance metrics? This article will show how easily PCP can help in these cases, with the help of pmda-openmetrics. OpenMetrics allows us to “turn numbers on the filesystem” into metrics in PCP.

What can OpenMetrics do?

PMDAs that come as part of RHEL can be listed with

$ yum search pcp-pmda

These have tested and proven code for reading metric data from often-used software, like databases, and the Linux system itself. If one of these PMDA fits your needs, then using that is usually best.

We assume in this article that our application is not covered by an existing PMDA, so we will use pmda-openmetrics, which is available in RHEL 8.2 (which includes PCP 5.0.2) and later. With OpenMetrics, PCP can easily cover the following situations:

• Metrics that are available in a text file, somewhere on the system

• Metrics that are available via URL, offered by some variant of web server (i.e. http://localhost/status)

• Metrics that are output to STDOUT by shell code which OpenMetrics executes

With these, the metrics become available in pmcd (so for live access), can be archived via pmlogger, and can be visualized, for example, using redis/grafana. 

A further option of monitoring custom applications is to write your own PMDA. PCP comes with the trivial and simple PMDAs which give a starting point for that. Such custom PMDAs are outside of the scope of Red Hat support, unless they get upstreamed and eventually find their way into RHEL. 

Application monitoring with OpenMetrics

We will implement an example monitoring where we have a script that  grabs metrics from our applications, and writes them into a text file. Then pmda-openmetrics grabs the data from the file. As this combination is based on a text file for metrics handover, debugging gets a lot easier.

On our test system, we have previously installed pcp-zeroconf, so a decent default PCP setup is in place. We will now install pcp-pmda-openmetrics (the PMDA we will use here), and bc as our script depends on it.

[root@космос ~]$ yum install pcp-pmda-openmetrics bc

[..]

[root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/

[root@космос openmetrics]$ ./Install

Let's say we are running a webserver and want to track the ratio of IPv6 connections vs. classic IPv4 over time. No native PMDA exists, but the values are rather easy to compute on the system. Our first component is the following shell script:

[chris@космос ~]$ cat bin/openmetrics_netstats.sh

#!/bin/bash

/usr/sbin/ss -tu|/usr/bin/egrep -v '127.0.0.1|Netid' >/tmp/sockets.txt

IPV6SOCKETS=$(grep -c '\[' /tmp/sockets.txt)

IPV4SOCKETS=$(grep -vc '\[' /tmp/sockets.txt)

IPV6RATE=$(echo "${IPV6SOCKETS}*100/(${IPV6SOCKETS}+${IPV4SOCKETS})"|bc)

IPV4RATE=$(echo "100-${IPV6RATE}"|bc)

echo "sockets {protocol=\"IPv6\"} ${IPV6RATE}"

echo "sockets {protocol=\"IPv4\"} ${IPV4RATE}"

[chris@космос ~]$ chmod +x bin/openmetrics_netstats.sh

[chris@космос ~]$ 

When executing this file, we get our metrics output to standard output. After executing crontab -e as a normal user on the system, we have set up this cronjob:

*/5 * * * * /home/chris/bin/openmetrics_netstats.sh >/tmp/openmetrics_netstat.txt

With this, we get the current IP6/v4 stats written to the text file in 5 min intervals. 

Now lets get the metrics grabbed by PCP. We create a configfile for OpenMetrics and tell it about our metrics file. The configurations in the config.d directory can be changed at any time since PCP picks them up on the fly.

[root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/config.d

[root@космос config.d]$ cat netstats.url

file:///tmp/openmetrics_netstat.txt

[root@космос config.d]$ 

At that point, the metrics become available in pmcd:

[chris@космос ~]$ pminfo openmetrics

openmetrics.netstats.sockets

[..]

[chris@космос ~]$ pmrep openmetrics.netstats

  o.n.sockets  o.n.sockets

  0 protocol:  1 protocol:

    25.000    75.000

    25.000    75.000

    25.000    75.000

So now we have verified that our data is available via pmcd. As you can see above, our config file netstats.url resulted in metric openmetrics.netstats. Effectively, the .url has been stripped off and the file name became the name of our new leaf: openmetrics.netstats. If we were directly running shell code here, with a file called netstatsshell.sh, our new leaf would be openmetrics.netstatsshell.

Archiving OpenMetrics data

Let’s also have our metric archived from now on--so in the future we can use the PCP archive files to understand how IPv6 has spread over time. On this system, we did initially install package pcp-zeroconf, which did already configure logging to archive files. 

Pmlogger is using the control-file to understand what should be archived: which metrics should be archived, how frequently, and so on. On RHEL 8.3 beta and later, the OpenMetrics metric should at this point already be included in our archive files, but let us verify. We will query for the currently active archive file, and search for the metric:

[root@космос ~]$ pcp

[..]

 pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.41

[chris@космос ~]$ pminfo \

  -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.41 openmetrics

[..]

Openmetrics.netstats.sockets

With this, we have verified that OpenMetrics appeared in the archive files!

If OpenMetrics was not found by the last command, we need to modify the control-file as follows:

[root@космос ~]$ vi /var/lib/pcp/config/pmlogger/config.default

[..]

# DO NOT UPDATE THE FILE ABOVE THIS LINE

# Otherwise any changes may be lost the next time pmlogconf is

# used on this file.

#

# It is safe to make additions from here on ...

#

log advisory on default {

    openmetrics

}

[..]

This section is at the end of the control-file. After restarting pmlogger, we again find out which archive we are currently logging to, and then search for metric openmetrics in the archive file:

[root@космос ~]$ systemctl restart pmlogger

[root@космос ~]$ pcp

[..]

 pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.43

[chris@космос ~]$ pminfo \

  -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.43 openmetrics

[..]

openmetrics.netstats.sockets

Our new metrics are getting archived!

Summary

With some simple steps, PCP can be configured to collect metrics which were not covered by the default PMDAs. Here, we used a text file as a buffer, but things get even simpler when executing shell code directly from OpenMetrics. If something can be expressed as numbers on the system, then it can also be taken over into PCP for monitoring.

Once the data is in PCP and archive files, we can use the full range of PCP tools: pmdiff to find peaks/unusual patterns in our data, redis/Grafana for visualization, and so on!

Details on visualizing PCP metrics with Grafana can be found in the product documentation.