Introduction to the Tracer feature in Satellite

Satellite 6.3 introduced a technology preview of a new feature named tracer. Tracer, still in tech preview in Satellite 6.4, helps administrators identify applications that need to be restarted after a Red Hat Enterprise Linux system is patched.  

For readers not familiar with what Red Hat technology previews are, when they should be used, and what level of support is provided, please review Red Hat’s Technology Preview Features Support Scope.

Tracer’s use cases

Some system administrators patch their systems, then simply check to see if the kernel was updated. If so, they reboot the server, and if not, they assume everything is good to go and that no reboot or application restarts are necessary. Unfortunately, it isn’t this simple, and this line of thinking may leave your systems vulnerable to security and bug issues that you thought were resolved, but in fact, are not.

Let’s look at an example where we run yum update and it reports that several MariaDB related packages need to be updated: mariadb, mariadb-libs, mariadb-server.

When we upgrade these packages, there is a RPM script in the mariadb-server RPM to run systemctl try-restart mariadb.service, which is supposed to restart the mariadb.service if it is running. This is good.  

However, other applications that depend on libraries provided in mariadb-libs are not automatically restarted, and will continue to use the old version of the libraries until they are restarted. This may be a major concern if the upgraded libraries fix a security vulnerability or critical bug, because it is very easy to think you have patched the servers and resolved the issue, when in fact you might still have running processes using the old version of the libraries.

In our example, we also have Postfix installed and running on the server, and one of Postfix’s dependencies is the mariadb-libs package. After we upgrade mariadb-libs, however, the running Postfix process will still be using the old version of the libraries provided by the mariadb-libs package.   

We can see this by finding the PID of the postfix process, then looking at the /proc/<PID>/maps file to see if there are any references to deleted files:

# ps -ef | grep postfix/master
root   4475 1  0 20:42 ?     00:00:00 /usr/libexec/postfix/master -w

In our example, the PID is 4475.  

​# cat /proc/4475/maps | grep -i deleted
7ff9e7b6b000-7ff9e7e49000 r-xp 00000000 fd:00 50360081                /usr/lib64/mysql/libmysqlclient.so.18.0.0;5b466c11 (deleted)
7ff9e7e49000-7ff9e8048000 ---p 002de000 fd:00 50360081                /usr/lib64/mysql/libmysqlclient.so.18.0.0;5b466c11 (deleted)
7ff9e8048000-7ff9e8051000 r--p 002dd000 fd:00 50360081                /usr/lib64/mysql/libmysqlclient.so.18.0.0;5b466c11 (deleted)
7ff9e8051000-7ff9e8065000 rw-p 002e6000 fd:00 50360081                /usr/lib64/mysql/libmysqlclient.so.18.0.0;5b466c11 (deleted)

We can see that there are several references to /usr/lib64/mysql/libmysqlclient.so.18.0.0;5b466c11, which is reported as deleted (due to the update on the mariadb-libs package).   

In this case, if we restart Postfix, it will start using the updated libraries, and we shouldn’t see any “deleted” references in the /proc/<PID>/maps file any longer:

# systemctl restart postfix
# ps -ef | grep postfix/master
root   7686 1  0 21:23 ?     00:00:00 /usr/libexec/postfix/master -w
# cat /proc/7686/maps | grep -i deleted
#

But how are system administrators supposed to know which applications should be restarted after updating the system? 

Tracer to the Rescue

The tracer utility helps you find outdated running applications in your system. To do this, it looks for applications that have any version of a file loaded in memory which was provided by any package updated since the system was booted up.   

Let’s try updating the three mariadb packages again (mariadb, mariadb-server, mariadb-libs), then see what the tracer command reports after the update:

# tracer
You should restart:
  * Some applications using:
      systemctl restart postfix

Tracer reports that postfix should be restarted, and even provides the command you can run to do so. If we run the suggested command, then run tracer again, it no longer reports any applications need to be restarted:

# systemctl restart postfix
# tracer
#

Tracer Integration in Satellite

As mentioned, Satellite 6.3 introduced tracer as a technology preview feature. To try out this technology preview, there aren’t any changes required on the Satellite server.  

Assuming you are using Satellite 6.3, on each Satellite client that will be using tracer, first make sure that the “rhel-7-server-satellite-tools-6.3-rpms” repository is enabled, then install the katello-host-tools-tracer package:

# yum install katello-host-tools-tracer

Next, run the katello-tracer-upload command, which will upload the tracer data to the Satellite server:

# katello-tracer-upload

To verify that Satellite is receiving tracer information from the client, within the Satellite web interface, go to “Hosts”, “All hosts”, and click on the name of the host that tracer was installed on.   

The host should now have a “Traces” property:

To see the details for the traces on the host, from the Satellite web interface, go to “Hosts”, “Content Hosts”, and click on the name of a host. Next, click on the “Traces” tab:

Let’s try the upgrade again on the three mariadb packages (mariadb, mariadb-server, mariadb-libs) and see what is reported in the Satellite web interface. We can either run a yum update on the client, or install the updates from Satellite. In this example, we will install the updates from the Satellite web interface:

Now that the packages are updated, if we go back to “Hosts”, “All hosts”, in the Satellite web interface, we can see that the host now has an orange indicator next to its name:

If we click on the host name, we can see more details:

It is reporting that “One or more processes require restarting”.  

If we go to “Hosts”, “Content Hosts” in the Satellite web interface, click on this host name, and go to the “Traces” tab we can see more details:

We can see that it is recommending that postfix should be restarted. If we click the box for postfix, then click “Restart Selected”, it will use Satellite remote execution to restart the postfix process:

Now that the postfix process has been restarted, the “Hosts”, “All Hosts” line for this host no longer shows the orange indicator, and instead, we have a green indicator.   

Final Thoughts

Tracer is an exciting technology preview feature in Satellite 6.3 which helps system administrators identify processes that need to be restarted after patching a host. This can help system administrators keep their systems patched and secured.