With Red Hat Enterprise Linux (RHEL) 8.2, we have extended the tools for visualizing performance data--so called “metrics”. These metrics can be collected by PCP, the Performance Co-Pilot framework. PCP comes with code to collect metrics from a wide area of commonly used software: Postgresql databases, systems temperature sensors, KVM guest stats, Apache httpd and so on; all of these can be monitored out of the box. These code pieces specialized in an area of monitoring are Performance Metric Domain Agents (PMDA), with package names pmda-apache, pmda-postgresql and so on.
But what if the RHEL repositories have no PMDA for an application? Our customers run all kinds of applications, so how to get an overview of their performance metrics? This article will show how easily PCP can help in these cases, with the help of pmda-openmetrics. OpenMetrics allows us to “turn numbers on the filesystem” into metrics in PCP.
What can OpenMetrics do?
PMDAs that come as part of RHEL can be listed with
$ yum search pcp-pmda
These have tested and proven code for reading metric data from often-used software, like databases, and the Linux system itself. If one of these PMDA fits your needs, then using that is usually best.
We assume in this article that our application is not covered by an existing PMDA, so we will use pmda-openmetrics, which is available in RHEL 8.2 (which includes PCP 5.0.2) and later. With OpenMetrics, PCP can easily cover the following situations:
-
Metrics that are available in a text file, somewhere on the system
-
Metrics that are available via URL, offered by some variant of web server (i.e. http://localhost/status)
-
Metrics that are output to STDOUT by shell code which OpenMetrics executes
With these, the metrics become available in pmcd (so for live access), can be archived via pmlogger, and can be visualized, for example, using redis/grafana.
A further option of monitoring custom applications is to write your own PMDA. PCP comes with the trivial
and simple
PMDAs which give a starting point for that. Such custom PMDAs are outside of the scope of Red Hat support, unless they get upstreamed and eventually find their way into RHEL.
Application monitoring with OpenMetrics
We will implement an example monitoring where we have a script that grabs metrics from our applications, and writes them into a text file. Then pmda-openmetrics grabs the data from the file. As this combination is based on a text file for metrics handover, debugging gets a lot easier.
On our test system, we have previously installed pcp-zeroconf, so a decent default PCP setup is in place. We will now install pcp-pmda-openmetrics (the PMDA we will use here), and bc
as our script depends on it.
[root@космос ~]$ yum install pcp-pmda-openmetrics bc [..] [root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/ [root@космос openmetrics]$ ./Install
Let's say we are running a webserver and want to track the ratio of IPv6 connections vs. classic IPv4 over time. No native PMDA exists, but the values are rather easy to compute on the system. Our first component is the following shell script:
[chris@космос ~]$ cat bin/openmetrics_netstats.sh #!/bin/bash /usr/sbin/ss -tu|/usr/bin/egrep -v '127.0.0.1|Netid' >/tmp/sockets.txt IPV6SOCKETS=$(grep -c '\[' /tmp/sockets.txt) IPV4SOCKETS=$(grep -vc '\[' /tmp/sockets.txt) IPV6RATE=$(echo "${IPV6SOCKETS}*100/(${IPV6SOCKETS}+${IPV4SOCKETS})"|bc) IPV4RATE=$(echo "100-${IPV6RATE}"|bc) echo "sockets {protocol=\"IPv6\"} ${IPV6RATE}" echo "sockets {protocol=\"IPv4\"} ${IPV4RATE}" [chris@космос ~]$ chmod +x bin/openmetrics_netstats.sh [chris@космос ~]$
When executing this file, we get our metrics output to standard output. After executing crontab -e
as a normal user on the system, we have set up this cronjob:
*/5 * * * * /home/chris/bin/openmetrics_netstats.sh >/tmp/openmetrics_netstat.txt
With this, we get the current IP6/v4 stats written to the text file in 5 min intervals.
Now lets get the metrics grabbed by PCP. We create a configfile for OpenMetrics and tell it about our metrics file. The configurations in the config.d directory can be changed at any time since PCP picks them up on the fly.
[root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/config.d [root@космос config.d]$ cat netstats.url file:///tmp/openmetrics_netstat.txt [root@космос config.d]$
At that point, the metrics become available in pmcd:
[chris@космос ~]$ pminfo openmetrics openmetrics.netstats.sockets [..] [chris@космос ~]$ pmrep openmetrics.netstats o.n.sockets o.n.sockets 0 protocol: 1 protocol: 25.000 75.000 25.000 75.000 25.000 75.000
So now we have verified that our data is available via pmcd. As you can see above, our config file netstats.url
resulted in metric openmetrics.netstats
. Effectively, the .url
has been stripped off and the file name became the name of our new leaf: openmetrics.netstats. If we were directly running shell code here, with a file called netstatsshell.sh
, our new leaf would be openmetrics.netstatsshell.
Archiving OpenMetrics data
Let’s also have our metric archived from now on--so in the future we can use the PCP archive files to understand how IPv6 has spread over time. On this system, we did initially install package pcp-zeroconf, which did already configure logging to archive files.
Pmlogger is using the control-file to understand what should be archived: which metrics should be archived, how frequently, and so on. On RHEL 8.3 beta and later, the OpenMetrics metric should at this point already be included in our archive files, but let us verify. We will query for the currently active archive file, and search for the metric:
[root@космос ~]$ pcp [..] pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.41 [chris@космос ~]$ pminfo \ -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.41 openmetrics [..] Openmetrics.netstats.sockets
With this, we have verified that OpenMetrics appeared in the archive files!
If OpenMetrics was not found by the last command, we need to modify the control-file as follows:
[root@космос ~]$ vi /var/lib/pcp/config/pmlogger/config.default [..] # DO NOT UPDATE THE FILE ABOVE THIS LINE # Otherwise any changes may be lost the next time pmlogconf is # used on this file. # # It is safe to make additions from here on ... # log advisory on default { openmetrics } [..]
This section is at the end of the control-file. After restarting pmlogger, we again find out which archive we are currently logging to, and then search for metric openmetrics
in the archive file:
[root@космос ~]$ systemctl restart pmlogger [root@космос ~]$ pcp [..] pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.43 [chris@космос ~]$ pminfo \ -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.43 openmetrics [..] openmetrics.netstats.sockets
Our new metrics are getting archived!
Summary
With some simple steps, PCP can be configured to collect metrics which were not covered by the default PMDAs. Here, we used a text file as a buffer, but things get even simpler when executing shell code directly from OpenMetrics. If something can be expressed as numbers on the system, then it can also be taken over into PCP for monitoring.
Once the data is in PCP and archive files, we can use the full range of PCP tools: pmdiff
to find peaks/unusual patterns in our data, redis/Grafana for visualization, and so on!
Details on visualizing PCP metrics with Grafana can be found in the product documentation.
執筆者紹介
Christian Horn is a Senior Technical Account Manager at Red Hat. After working with customers and partners since 2011 at Red Hat Germany, he moved to Japan, focusing on mission critical environments. Virtualization, debugging, performance monitoring and tuning are among the returning topics of his daily work. He also enjoys diving into new technical topics, and sharing the findings via documentation, presentations or articles.
チャンネル別に見る
自動化
テクノロジー、チームおよび環境に関する IT 自動化の最新情報
AI (人工知能)
お客様が AI ワークロードをどこでも自由に実行することを可能にするプラットフォームについてのアップデート
オープン・ハイブリッドクラウド
ハイブリッドクラウドで柔軟に未来を築く方法をご確認ください。
セキュリティ
環境やテクノロジー全体に及ぶリスクを軽減する方法に関する最新情報
エッジコンピューティング
エッジでの運用を単純化するプラットフォームのアップデート
インフラストラクチャ
世界有数のエンタープライズ向け Linux プラットフォームの最新情報
アプリケーション
アプリケーションの最も困難な課題に対する Red Hat ソリューションの詳細
オリジナル番組
エンタープライズ向けテクノロジーのメーカーやリーダーによるストーリー
製品
ツール
試用、購入、販売
コミュニケーション
Red Hat について
エンタープライズ・オープンソース・ソリューションのプロバイダーとして世界をリードする Red Hat は、Linux、クラウド、コンテナ、Kubernetes などのテクノロジーを提供しています。Red Hat は強化されたソリューションを提供し、コアデータセンターからネットワークエッジまで、企業が複数のプラットフォームおよび環境間で容易に運用できるようにしています。
言語を選択してください
Red Hat legal and privacy links
- Red Hat について
- 採用情報
- イベント
- 各国のオフィス
- Red Hat へのお問い合わせ
- Red Hat ブログ
- ダイバーシティ、エクイティ、およびインクルージョン
- Cool Stuff Store
- Red Hat Summit