Overcoming vulnerabilities with live kernel patching in Red Hat Enterprise Linux 8.5

IT operations folks strive to not only maximize uptime, but also keep systems patched. These might seem like competing goals, but we're here to help with Live kernel patching in Red Hat Enterprise Linux (RHEL) and enjoy some improvements with live kernel patching in RHEL 8.5.

You can use a built-in tool to get update the kernel on RHEL systems with no downtime. That tool is live kernel patching (kpatch). Kpatch has been a part of our operating system for some time now (since RHEL 8.1, 7.7). However, with RHEL 8.5 (and the 9.0 Beta), there are some significant enhancements.

What is live kernel patching?

Suppose you have worked as a Systems Administrator or in a similar role. If so, you are likely familiar with this scenario: You get to the office and scroll through your emails and you see an urgent message about a new CVE. The clock has already started ticking.

According to company policy, Operations must remediate all critical vulnerabilities within 48 hours. However, that same policy doesn't allow you to take downtime more than once a month, usually on a Saturday night and not until 2 AM. (Sounding familiar yet?) What is an Administrator to do?

That is where live kernel patching comes into play: Upstream engineers write a fix for an appropriate CVE and release it in a new Linux kernel. Red Hat backports this fix into the RHEL kernel and delivers it as a standard update (RPM) and a Live Kernel Patch package, both made available through the official Red Hat repositories. When you install the Live Kernel Patch onto the target RHEL system, the new kernel RPM is installed and the patch is automatically applied to the running kernel, live, without the need to reboot. You’ve addressed the CVE and not violated your uptime requirements! 

As an added advantage, these live patches are cumulative. That means, when a live kernel patch is applied, it includes the fixes from within the current lookback window (details in the next section).  These cumulative patches allow multiple issues to be fixed with a single action on a running live kernel and leave the administrator free to reboot the system in question at a later time, typically as part of a planned maintenance activity.

What is new with live kernel patching?

In the previous section, we mentioned that patches are cumulative. When first introduced in Red Hat Enterprise Linux, Live Kernel Patches were only available for Extended Update Support (EUS) releases. (As a general rule, this is every even-numbered minor release. I.e. 8.2, 8.4, etc.) These live patches would then span a 12-month look back, the window in which fixes accumulated.

This policy provided a long runway for implementing critical fixes and a large window in which to perform planned reboots. However, for organizations looking to iterate quickly, the drawback was every other minor release would be supported. This limitation slowed down development teams who benefited from leveraging the latest versions of languages, tools and libraries released in each minor version of RHEL 8.

As of RHEL 8.5 and 9.0 Beta, the lookback window has been adjusted. Now, each minor release will have support for kpatch with a 6-month lookback. This new policy makes it easier to stay on the latest supported version of RHEL, get the latest in performance improvements and developer tool sets, and still provide the ability to address known CVEs, while being able to more directly control what downtime is required for patching.

With the increased availability of live kernel patches, we have also included functionality to make this process even easier! While today, Live Kernel Patches are easily installed via the command line, in the latest release there is support for this same operation through the web console! This new capability brings greater visibility and accessibility to live kernel patching by placing this information right into the same tool you use to administrate your RHEL systems today, the Web Console.

How do you use live kernel patching?

If you are running an instance of Red Hat Enterprise Linux newer than 7.7 or 8.1, you can take advantage of the kpatch service:

# yum install -y kpatch
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 25 MB/s | 37 MB 00:01   
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 42 MB/s | 41 MB 00:00    
Dependencies resolved.
=========================================================================================================================
 Package Architecture Version Repository Size
=========================================================================================================================
Installing:
 kpatch noarch 0.9.2-5.el8 rhel-8-for-x86_64-baseos-rpms 17 k
Installing weak dependencies:
 kpatch-dnf noarch 0.2-5.el8 rhel-8-for-x86_64-baseos-rpms 17 k
<<< OUTPUT TRUNCATED >>>

With the kpatch package installed, we can see what patches are available. The first command will let us know what the running kernel version is. The second command will list the available patches for that kernel:

# uname -r
4.18.0-305.el8.x86_64
# yum list available kpatch-patch*305*
Updating Subscription Management repositories.
Last metadata expiration check: 0:02:28 ago on Thu 02 Dec 2021 12:31:59 AM EST.
Available Packages
kpatch-patch-4_18_0-305.x86_64 1-6.el8 rhel-8-for-x86_64-baseos-rpms
kpatch-patch-4_18_0-305_10_2.x86_64 1-3.el8_4 rhel-8-for-x86_64-baseos-rpms
kpatch-patch-4_18_0-305_12_1.x86_64 1-2.el8_4 rhel-8-for-x86_64-baseos-rpms
kpatch-patch-4_18_0-305_17_1.x86_64 1-1.el8_4 <<< OUTPUT TRUNCATED >>>

Either we install specific packages from this list, or we can embed our kernel into the yum install command to make it easier to install the most appropriate package:

# yum install -y "kpatch-patch = $(uname -r)"
Updating Subscription Management repositories.
Last metadata expiration check: 0:04:23 ago on Thu 02 Dec 2021 12:31:59 AM EST.
Dependencies resolved.
=========================================================================================================================
 Package Architecture Version Repository Size
=========================================================================================================================
Installing:
 kpatch-patch-4_18_0-305 x86_64 1-6.el8 rhel-8-for-x86_64-baseos-rpms 49 k
<<< OUTPUT TRUNCATED>>>

Finally, let's verify that we have successfully applied the Live Kernel Patch:

# kpatch list
Loaded patch modules:
kpatch_4_18_0_305_1_6 [enabled]

Installed patch modules:
kpatch_4_18_0_305_1_6 (4.18.0-305.el8.x86_64)

Just like that, we have installed and applied a Live Kernel Patch for a vulnerability in our running system. This server can reboot later when we aren't in the middle of an urgent security activity.

The process shown above is even more straightforward inside of the Web UI! Log in and navigate to the Software Updates task in the left-hand menu.

From there, configure Kernel patch settings in the right-hand box. That's it. Your system is ready to run with live kernel patches! You can either enable automatic updates or install the live kernel patches independently.

Want to try this for yourself? Go to our lab for live kernel patching. Get more details, try this out on a live RHEL instance, and best of all no need to sign up or even log in.

Wrap up

Live kernel patching provides that next-level of operational efficiency for the busy operations teams to support infrastructure running critical workloads when downtime is not an option, and security responsiveness is required. Without rebooting a RHEL instance, you can now install critical fixes on the running kernel. While this only applies to the kernel today, the long-term vision is to support a growing list of services and packages too!

If you would like to read more about live kernel patching, please take a look at our documentation: Managing, monitoring, and updating the kernel Red Hat Enterprise Linux 8 | Red Hat Customer Portal.

If you are new to Red Hat Enterprise Linux and are excited about running an enterprise operating system that has live kernel patching capabilities, get started today by visiting our page to download the Red Hat Enterprise Linux Server free 60-day trial!