With Red Hat Enterprise Linux (RHEL) 8.2, we have extended the tools for visualizing performance data--so called “metrics”. These metrics can be collected by PCP, the Performance Co-Pilot framework. PCP comes with code to collect metrics from a wide area of commonly used software: Postgresql databases, systems temperature sensors, KVM guest stats, Apache httpd and so on; all of these can be monitored out of the box. These code pieces specialized in an area of monitoring are Performance Metric Domain Agents (PMDA), with package names pmda-apache, pmda-postgresql and so on.
But what if the RHEL repositories have no PMDA for an application? Our customers run all kinds of applications, so how to get an overview of their performance metrics? This article will show how easily PCP can help in these cases, with the help of pmda-openmetrics. OpenMetrics allows us to “turn numbers on the filesystem” into metrics in PCP.
What can OpenMetrics do?
PMDAs that come as part of RHEL can be listed with
$ yum search pcp-pmda
These have tested and proven code for reading metric data from often-used software, like databases, and the Linux system itself. If one of these PMDA fits your needs, then using that is usually best.
We assume in this article that our application is not covered by an existing PMDA, so we will use pmda-openmetrics, which is available in RHEL 8.2 (which includes PCP 5.0.2) and later. With OpenMetrics, PCP can easily cover the following situations:
-
Metrics that are available in a text file, somewhere on the system
-
Metrics that are available via URL, offered by some variant of web server (i.e. http://localhost/status)
-
Metrics that are output to STDOUT by shell code which OpenMetrics executes
With these, the metrics become available in pmcd (so for live access), can be archived via pmlogger, and can be visualized, for example, using redis/grafana.
A further option of monitoring custom applications is to write your own PMDA. PCP comes with the trivial
and simple
PMDAs which give a starting point for that. Such custom PMDAs are outside of the scope of Red Hat support, unless they get upstreamed and eventually find their way into RHEL.
Application monitoring with OpenMetrics
We will implement an example monitoring where we have a script that grabs metrics from our applications, and writes them into a text file. Then pmda-openmetrics grabs the data from the file. As this combination is based on a text file for metrics handover, debugging gets a lot easier.
On our test system, we have previously installed pcp-zeroconf, so a decent default PCP setup is in place. We will now install pcp-pmda-openmetrics (the PMDA we will use here), and bc
as our script depends on it.
[root@космос ~]$ yum install pcp-pmda-openmetrics bc [..] [root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/ [root@космос openmetrics]$ ./Install
Let's say we are running a webserver and want to track the ratio of IPv6 connections vs. classic IPv4 over time. No native PMDA exists, but the values are rather easy to compute on the system. Our first component is the following shell script:
[chris@космос ~]$ cat bin/openmetrics_netstats.sh #!/bin/bash /usr/sbin/ss -tu|/usr/bin/egrep -v '127.0.0.1|Netid' >/tmp/sockets.txt IPV6SOCKETS=$(grep -c '\[' /tmp/sockets.txt) IPV4SOCKETS=$(grep -vc '\[' /tmp/sockets.txt) IPV6RATE=$(echo "${IPV6SOCKETS}*100/(${IPV6SOCKETS}+${IPV4SOCKETS})"|bc) IPV4RATE=$(echo "100-${IPV6RATE}"|bc) echo "sockets {protocol=\"IPv6\"} ${IPV6RATE}" echo "sockets {protocol=\"IPv4\"} ${IPV4RATE}" [chris@космос ~]$ chmod +x bin/openmetrics_netstats.sh [chris@космос ~]$
When executing this file, we get our metrics output to standard output. After executing crontab -e
as a normal user on the system, we have set up this cronjob:
*/5 * * * * /home/chris/bin/openmetrics_netstats.sh >/tmp/openmetrics_netstat.txt
With this, we get the current IP6/v4 stats written to the text file in 5 min intervals.
Now lets get the metrics grabbed by PCP. We create a configfile for OpenMetrics and tell it about our metrics file. The configurations in the config.d directory can be changed at any time since PCP picks them up on the fly.
[root@космос ~]$ cd /var/lib/pcp/pmdas/openmetrics/config.d [root@космос config.d]$ cat netstats.url file:///tmp/openmetrics_netstat.txt [root@космос config.d]$
At that point, the metrics become available in pmcd:
[chris@космос ~]$ pminfo openmetrics openmetrics.netstats.sockets [..] [chris@космос ~]$ pmrep openmetrics.netstats o.n.sockets o.n.sockets 0 protocol: 1 protocol: 25.000 75.000 25.000 75.000 25.000 75.000
So now we have verified that our data is available via pmcd. As you can see above, our config file netstats.url
resulted in metric openmetrics.netstats
. Effectively, the .url
has been stripped off and the file name became the name of our new leaf: openmetrics.netstats. If we were directly running shell code here, with a file called netstatsshell.sh
, our new leaf would be openmetrics.netstatsshell.
Archiving OpenMetrics data
Let’s also have our metric archived from now on--so in the future we can use the PCP archive files to understand how IPv6 has spread over time. On this system, we did initially install package pcp-zeroconf, which did already configure logging to archive files.
Pmlogger is using the control-file to understand what should be archived: which metrics should be archived, how frequently, and so on. On RHEL 8.3 beta and later, the OpenMetrics metric should at this point already be included in our archive files, but let us verify. We will query for the currently active archive file, and search for the metric:
[root@космос ~]$ pcp [..] pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.41 [chris@космос ~]$ pminfo \ -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.41 openmetrics [..] Openmetrics.netstats.sockets
With this, we have verified that OpenMetrics appeared in the archive files!
If OpenMetrics was not found by the last command, we need to modify the control-file as follows:
[root@космос ~]$ vi /var/lib/pcp/config/pmlogger/config.default [..] # DO NOT UPDATE THE FILE ABOVE THIS LINE # Otherwise any changes may be lost the next time pmlogconf is # used on this file. # # It is safe to make additions from here on ... # log advisory on default { openmetrics } [..]
This section is at the end of the control-file. After restarting pmlogger, we again find out which archive we are currently logging to, and then search for metric openmetrics
in the archive file:
[root@космос ~]$ systemctl restart pmlogger [root@космос ~]$ pcp [..] pmlogger: primary logger: /var/log/pcp/pmlogger/kosmos/20200818.10.43 [chris@космос ~]$ pminfo \ -a /var/log/pcp/pmlogger/fluxcoil/20200818.10.43 openmetrics [..] openmetrics.netstats.sockets
Our new metrics are getting archived!
Summary
With some simple steps, PCP can be configured to collect metrics which were not covered by the default PMDAs. Here, we used a text file as a buffer, but things get even simpler when executing shell code directly from OpenMetrics. If something can be expressed as numbers on the system, then it can also be taken over into PCP for monitoring.
Once the data is in PCP and archive files, we can use the full range of PCP tools: pmdiff
to find peaks/unusual patterns in our data, redis/Grafana for visualization, and so on!
Details on visualizing PCP metrics with Grafana can be found in the product documentation.
저자 소개
Christian Horn is a Senior Technical Account Manager at Red Hat. After working with customers and partners since 2011 at Red Hat Germany, he moved to Japan, focusing on mission critical environments. Virtualization, debugging, performance monitoring and tuning are among the returning topics of his daily work. He also enjoys diving into new technical topics, and sharing the findings via documentation, presentations or articles.
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.