With Red Hat Enterprise Linux (RHEL) 8.2, we have extended the tools for visualizing performance data--so called “metrics”. These metrics can be collected by PCP, the Performance Co-Pilot framework. PCP comes with code to collect metrics from a wide area of commonly used software: Postgresql databases, systems temperature sensors, KVM guest stats, Apache httpd and so on; all of these can be monitored out of the box. These code pieces specialized in an area of monitoring are Performance Metric Domain Agents (PMDA), with package names pmda-apache, pmda-postgresql and so on.
But what if the RHEL repositories have no PMDA for an application? Our customers run all kinds of applications, so how to get an overview of their performance metrics? This article will show how easily PCP can help in these cases, with the help of pmda-openmetrics. OpenMetrics allows us to “turn numbers on the filesystem” into metrics in PCP.
What can OpenMetrics do?
PMDAs that come as part of RHEL can be listed with
$ yum search pcp-pmda
These have tested and proven code for reading metric data from often-used software, like databases, and the Linux system itself. If one of these PMDA fits your needs, then using that is usually best.
We assume in this article that our application is not covered by an existing PMDA, so we will use pmda-openmetrics, which is available in RHEL 8.2 (which includes PCP 5.0.2) and later. With OpenMetrics, PCP can easily cover the following situations:
-
Metrics that are available in a text file, somewhere on the system
-
Metrics that are available via URL, offered by some variant of web server (i.e. http://localhost/status)
-
Metrics that are output to STDOUT by shell code which OpenMetrics executes
With these, the metrics become available in pmcd (so for live access), can be archived via pmlogger, and can be visualized, for example, using redis/grafana.
A further option of monitoring custom applications is to write your own PMDA. PCP comes with the trivial
and simple
PMDAs which give a starting point for that. Such custom PMDAs are outside of the scope of Red Hat support, unless they get upstreamed and eventually find their way into RHEL.
Application monitoring with OpenMetrics
We will implement an example monitoring where we have a script that grabs metrics from our applications, and writes them into a text file. Then pmda-openmetrics grabs the data from the file. As this combination is based on a text file for metrics handover, debugging gets a lot easier.
On our test system, we have previously installed pcp-zeroconf, so a decent default PCP setup is in place. We will now install pcp-pmda-openmetrics (the PMDA we will use here), and bc
as our script depends on it.
[root@космос ~]$ yum install pcp-pmda-openmetrics bc [..] [root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/ [root@космос openmetrics]$ ./Install
Let's say we are running a webserver and want to track the ratio of IPv6 connections vs. classic IPv4 over time. No native PMDA exists, but the values are rather easy to compute on the system. Our first component is the following shell script:
[chris@космос ~]$ cat bin/openmetrics_netstats.sh #!/bin/bash /usr/sbin/ss -tu|/usr/bin/egrep -v '127.0.0.1|Netid' >/tmp/sockets.txt IPV6SOCKETS=$(grep -c '\[' /tmp/sockets.txt) IPV4SOCKETS=$(grep -vc '\[' /tmp/sockets.txt) IPV6RATE=$(echo "${IPV6SOCKETS}*100/(${IPV6SOCKETS}+${IPV4SOCKETS})"|bc) IPV4RATE=$(echo "100-${IPV6RATE}"|bc) echo "sockets {protocol=\"IPv6\"} ${IPV6RATE}" echo "sockets {protocol=\"IPv4\"} ${IPV4RATE}" [chris@космос ~]$ chmod +x bin/openmetrics_netstats.sh [chris@космос ~]$
When executing this file, we get our metrics output to standard output. After executing crontab -e
as a normal user on the system, we have set up this cronjob:
*/5 * * * * /home/chris/bin/openmetrics_netstats.sh >/tmp/openmetrics_netstat.txt
With this, we get the current IP6/v4 stats written to the text file in 5 min intervals.
Now lets get the metrics grabbed by PCP. We create a configfile for OpenMetrics and tell it about our metrics file. The configurations in the config.d directory can be changed at any time since PCP picks them up on the fly.
[root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/config.d [root@космос config.d]$ cat netstats.url file:///tmp/openmetrics_netstat.txt [root@космос config.d]$
At that point, the metrics become available in pmcd:
[chris@космос ~]$ pminfo openmetrics openmetrics.netstats.sockets [..] [chris@космос ~]$ pmrep openmetrics.netstats o.n.sockets o.n.sockets 0 protocol: 1 protocol: 25.000 75.000 25.000 75.000 25.000 75.000
So now we have verified that our data is available via pmcd. As you can see above, our config file netstats.url
resulted in metric openmetrics.netstats
. Effectively, the .url
has been stripped off and the file name became the name of our new leaf: openmetrics.netstats. If we were directly running shell code here, with a file called netstatsshell.sh
, our new leaf would be openmetrics.netstatsshell.
Archiving OpenMetrics data
Let’s also have our metric archived from now on--so in the future we can use the PCP archive files to understand how IPv6 has spread over time. On this system, we did initially install package pcp-zeroconf, which did already configure logging to archive files.
Pmlogger is using the control-file to understand what should be archived: which metrics should be archived, how frequently, and so on. On RHEL 8.3 beta and later, the OpenMetrics metric should at this point already be included in our archive files, but let us verify. We will query for the currently active archive file, and search for the metric:
[root@космос ~]$ pcp [..] pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.41 [chris@космос ~]$ pminfo \ -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.41 openmetrics [..] Openmetrics.netstats.sockets
With this, we have verified that OpenMetrics appeared in the archive files!
If OpenMetrics was not found by the last command, we need to modify the control-file as follows:
[root@космос ~]$ vi /var/lib/pcp/config/pmlogger/config.default [..] # DO NOT UPDATE THE FILE ABOVE THIS LINE # Otherwise any changes may be lost the next time pmlogconf is # used on this file. # # It is safe to make additions from here on ... # log advisory on default { openmetrics } [..]
This section is at the end of the control-file. After restarting pmlogger, we again find out which archive we are currently logging to, and then search for metric openmetrics
in the archive file:
[root@космос ~]$ systemctl restart pmlogger [root@космос ~]$ pcp [..] pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.43 [chris@космос ~]$ pminfo \ -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.43 openmetrics [..] openmetrics.netstats.sockets
Our new metrics are getting archived!
Summary
With some simple steps, PCP can be configured to collect metrics which were not covered by the default PMDAs. Here, we used a text file as a buffer, but things get even simpler when executing shell code directly from OpenMetrics. If something can be expressed as numbers on the system, then it can also be taken over into PCP for monitoring.
Once the data is in PCP and archive files, we can use the full range of PCP tools: pmdiff
to find peaks/unusual patterns in our data, redis/Grafana for visualization, and so on!
Details on visualizing PCP metrics with Grafana can be found in the product documentation.
Sull'autore
Christian Horn is a Senior Technical Account Manager at Red Hat. After working with customers and partners since 2011 at Red Hat Germany, he moved to Japan, focusing on mission critical environments. Virtualization, debugging, performance monitoring and tuning are among the returning topics of his daily work. He also enjoys diving into new technical topics, and sharing the findings via documentation, presentations or articles.
Ricerca per canale
Automazione
Novità sull'automazione IT di tecnologie, team e ambienti
Intelligenza artificiale
Aggiornamenti sulle piattaforme che consentono alle aziende di eseguire carichi di lavoro IA ovunque
Hybrid cloud open source
Scopri come affrontare il futuro in modo più agile grazie al cloud ibrido
Sicurezza
Le ultime novità sulle nostre soluzioni per ridurre i rischi nelle tecnologie e negli ambienti
Edge computing
Aggiornamenti sulle piattaforme che semplificano l'operatività edge
Infrastruttura
Le ultime novità sulla piattaforma Linux aziendale leader a livello mondiale
Applicazioni
Approfondimenti sulle nostre soluzioni alle sfide applicative più difficili
Serie originali
Raccontiamo le interessanti storie di leader e creatori di tecnologie pensate per le aziende
Prodotti
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Servizi cloud
- Scopri tutti i prodotti
Strumenti
- Formazione e certificazioni
- Il mio account
- Supporto clienti
- Risorse per sviluppatori
- Trova un partner
- Red Hat Ecosystem Catalog
- Calcola il valore delle soluzioni Red Hat
- Documentazione
Prova, acquista, vendi
Comunica
- Contatta l'ufficio vendite
- Contatta l'assistenza clienti
- Contatta un esperto della formazione
- Social media
Informazioni su Red Hat
Red Hat è leader mondiale nella fornitura di soluzioni open source per le aziende, tra cui Linux, Kubernetes, container e soluzioni cloud. Le nostre soluzioni open source, rese sicure per un uso aziendale, consentono di operare su più piattaforme e ambienti, dal datacenter centrale all'edge della rete.
Seleziona la tua lingua
Red Hat legal and privacy links
- Informazioni su Red Hat
- Opportunità di lavoro
- Eventi
- Sedi
- Contattaci
- Blog di Red Hat
- Diversità, equità e inclusione
- Cool Stuff Store
- Red Hat Summit