RHEL package updates and live kernel patching with Red Hat Satellite

In many environments, scheduling downtime to patch and reboot Red Hat Enterprise Linux (RHEL) systems is difficult, which can lead to systems being left vulnerable for weeks or months on end. Some of these security vulnerabilities are critical in nature.

RHEL versions 7.7 and 8.1 introduced live kernel patching functionality via kpatch for all subscriptions, which allows for select critical and important security kernel patches to be applied without a reboot. There are several considerations to take into account when using kpatch, so please review the relevant RHEL 7 and RHEL 8 documentation which covers kpatch in depth, including its limitations, before proceeding further. 

If the goal is to live patch a system for security vulnerabilities, we need more than kpatch, as it only handles kernel patches. We also need to consider other packages that might need to be updated on the system for security issues. One frequent mistake system administrators make is updating a system, but not restarting processes on the system that have had their libraries updated, which can lead to processes remaining vulnerable. 

Let’s look at an example where we run yum update and it reports that several MariaDB related packages need to be updated: mariadb, mariadb-libs, mariadb-server.

When we upgrade these packages, there is an RPM script in the mariadb-server RPM to run systemctl try-restart mariadb.service, which will restart the mariadb.service if it is running. However, other applications that depend on libraries provided in mariadb-libs are not automatically restarted, and will continue to use the old version of the libraries until they are restarted. This may be a major concern if the upgraded libraries fix a security vulnerability or critical bug, because it is very easy to think you have patched the servers and resolved the issue, when in fact you might still have running processes using the old version of the libraries.

In our example, we also have Postfix installed and running on the server, and one of Postfix’s dependencies is the mariadb-libs package. After we upgrade mariadb-libs, however, the running Postfix process will still be using the old version of the libraries provided by the mariadb-libs package. 

This is where the tracer feature of Satellite comes in to assist. Tracer helps administrators identify applications that need to be restarted after a RHEL system is patched, such as Postfix in our previous example. The tracer functionality, included in Satellite versions 6.3 through 6.6, is a technology preview feature.

In this post, I will be covering live patching a RHEL server for security updates with a combination of kpatch for live kernel patches, and tracer to identify applications that need to be restarted. 

Identify Kernel Versions in Environment

The first step to live patching kernels on systems in Satellite is to understand which versions of the kernel are running on Satellite client systems. This can be determined by logging in to the Satellite web interface, going to the Monitor menu, and selecting Facts. Next, for the filter, enter the uname::release fact name, which should show a list of the Satellite clients and their kernel versions, as shown:

The View Chart button on the right will bring up a chart showing the client kernel versions:

In this example, there are two servers running kernel 3.10.0-1062.el7.x86_64 (both are RHEL 7.7 servers), and one server running 2.6.32-696.el6.x86_64 (a RHEL 6 server). RHEL 6 does not support live kernel patching, so only the RHEL 7.7 servers will be able to install live kernel updates. 

After identifying the kernel version in the environment we want to live patch, the next step is to create a search bookmark for this kernel version we will be able to use later when applying updates. To do so, select the Hosts menu, and click on All Hosts. Next, filter for facts.uname::release = 3.10.0-1062.el7.x86_64 which will list clients at this kernel version. Click the bookmark icon dropdown menu, and select Bookmark this search:

In this example, I named the bookmark hosts with kernel 3.10.0-1062.el7.x86_64.

Identify available kpatch packages

Now that we have identified the kernel version we would like to live patch (in this example kernel 3.10.0-1062.el7.x86_64), we need to identify the kpatch updates available for this kernel version. 

The kpatch package naming convention is kpatch-patch-kernel_version-kpatch_level.os.arch, where kernel_version is the version of the kernel with all periods replaced with underscores. For example, the kernel version we are working with is 3.10.0-1062, which will be referred to as 3_10_0-1062 in the kpatch package file name. The kpatch_level in the naming convention refers to the version and release of the kpatch. A kpatch with a 0-0 level is the initial empty patch, used to subscribe the system to future live updates for this kernel version. A kpatch package with a 1-1 version or higher contains live kernel patches. The kpatch patches are cumulative, so in general, we want to use the latest one for the kernel version that is being live patched. 

To see the kpatch updates available in the Satellite environment,from the Satellite web interface, go to the Content menu, and select Packages. Next, filter for kpatch-patch, which will show a list of available kpatch live kernel update patches. 

From the previous screenshot, we can see that there are kpatch updates available for three different kernel versions: 3.10.0-1062 (green boxes), 3.10.0-1062.1.1 (blue boxes), and 3.10.0-1062.1.2 (purple boxes). 

In this post, we are working with kernel 3.10.0-1062, which are the 3 lines represented with the green boxes. For these lines, we can see that there are three kpatch levels available: 0-0, 1-1, and 1-2. Because kpatchs are cumulative, we will go with the latest kpatch version available for this kernel version, which is version 1-2. In this example, we’ve identified that we want to use kpatch-patch-3_10_0-1062-1-2.el7.x86_64. Verify that this package is present in the Content Views and Lifecycle Environments for the servers that will be updated, and publish/promote Content Views, if needed. 

Installing the live kernel patch with Satellite

We are now ready to install the live kernel patch on our Satellite clients running the 3.10.0-1062.el7.x86_64 kernel. To start the process, go to the Monitor menu, select Jobs, and click the blue Run Job button in the upper right corner. 

Next, select Ansible Packages for the job category and Package Action- Ansible Default for the job template. Additionally, select the previously created search bookmark named hosts with kernel 3.10.0-1062.el7.x86_64, which will automatically populate the search query. Leave the state option set to present and, for the name, put in the previously identified kpatch-patch-3_10_0-1062-1-2.el7.x86_64 package. Once we’ve completed these steps, click submit. 

The job will run and Satellite will report the results.

Verifying live kernel patch status on clients

We can verify the live kernel patches were successfully loaded by running another remote execution job, which will check the status of the kpatch list command. To check this, go back to the Monitor menu, select Jobs, and click the blue Run Job button in the upper right corner. 

This time, we will use the Ansible Commands job category and the Run Command - Ansible Default job template. We will again select the previously created search bookmark named hosts with kernel 3.10.0-1062.el7.x86_64. For the command, we will specify kpatch list | fgrep 'kpatch_3_10_0_1062_1_2 [enabled]’ Note that the text used for the fgrep command (kpatch_3_10_0_1062_1_2 [enabled]) doesn’t match the name of the package (kpatch-patch-3_10_0-1062-1-2.el7.x86_64). The easiest way to find the text to use in the command is to login to one of the client servers that was patched and run kpatch list to see a list of enabled kpatchs. 

This remote execution job will succeed on clients that have the kpatch loaded and fail for those that don’t. If this remote execution job fails on any hosts, investigate further to determine the issue. 

Viewing what CVE’s are included in live kernel patch

We can view which CVE’s were included in the live kernel patch by logging in to one of the patched clients, and running rpm -q --changelog kpatch-patch-3_10_0-1062-1-2.el7.x86_64.

[root@client1 ~]# rpm -q --changelog kpatch-patch-3_10_0-1062-1-2.el7.x86_64
* Tue Sep 17 2019 Artem Savkov <asavkov@redhat.com> [1-2.el7]
- brcmfmac heap buffer overflow in brcmf_wowl_nd_results [1751256] {CVE-2019-9500}

* Mon Sep 16 2019 Artem Savkov <asavkov@redhat.com> [1-1.el7]
- vhost-net: guest to host kernel escape during migration [1750892] {CVE-2019-14835}

* Fri Jul 19 2019 Artem Savkov <asavkov@redhat.com> [0-0.el7]
- An empty patch to subscribe to kpatch stream for kernel-3.10.0-1062.el7 [1730784]

Installing security updates for other packages

At this point, we’ve installed the live kernel patch, however, there might be other packages on the Satellite clients that need security related updates. This is where tracer comes in to help identify which processes need to be restarted after installing package updates. As a reminder, tracer, in Satellite versions 6.3 - 6.6, is a technology preview feature. 

Please review my previous post covering an Introduction to the Tracer feature in Satellite, as it covers how to get tracer setup in the environment. Having tracer setup in the environment is necessary before proceeding with further steps in this post. 

To install security related package updates, we’ll run another remote execution job by accessing the Monitor menu, selecting Jobs, and clicking the blue Run Job button in the upper right corner. We’ll use the Ansible Commands job category, and the Run Command - Ansible Default job template. As before, we’ll select the previously created search bookmark named hosts with kernel 3.10.0-1062.el7.x86_64. For the command, we’ll specify yum -y update --security --exclude=kernel --exclude=kernel-tools --exclude=kernel-tools-libs. This command will install the security related updates and exclude the kernel related packages, since we previously live patched the kernel. 

After the security updates have completed, we’ll utilize tracer to see if there are processes that need to be restarted for the updated packages to take effect. 

To check this, go to the Hosts menu, and click on Content Hosts. Next, click on a host that was updated and go to the Traces tab, which shows that the ksmtuned daemon needs to be restarted for the updated packages to take effect:

Depending on the environment, we might need to see which processes need to be restarted by checking each content host individually, selecting the processes to be restarted, and clicking the Restart Selected button. 

If you have a standard OS configuration where all the servers have the same packages installed, and the same updates were installed across all the servers, then it is likely that the same processes will need to be restarted on all the servers. In this case, we can check one of the content hosts to see which process(es) need to be restarted, make a note of the helper command to restart the process, then run a remote execution command. For the job category we will specify Katello, and for the job template specify Restart Services - Katello SSH Default. If appropriate, we can use the previously saved bookmark, and copy and paste the previously identified helper command, as shown:

After restarting the processes, it is recommended to check the status of the hosts under Hosts, All Hosts, to verify that tracer is not reporting any additional processes need to be restarted. 

Next Steps

For customers with standard subscriptions, Red Hat provides live kernel patches until the next minor version of RHEL is released. For example, Red Hat will provide live kernel patches for RHEL 8.1 until RHEL 8.2 is released. For customers with Extended Update Support subscriptions, Red Hat provides live kernel patches for one year from the date that the kernel was released. 

In either case, at some point, you will need to schedule downtime to update and reboot the RHEL servers. At that time, you will likely want to update all packages, including the kernel related packages, and then reboot. Once the servers have rebooted and are running on the updated kernel, the previously installed kpatch-patch (in our example, kpatch-patch-3_10_0-1062-1-2.el7.x86_64) should be uninstalled.  

Summary and Closing

In summary, using Satellite to install kernel and other security updates in RHEL 7 or RHEL 8 environments without rebooting can be useful. This is particularly pertinent in environments where reboots are difficult to schedule.