Working with the real-time kernel for Red Hat Enterprise Linux
The real-time kernel capability has existed for more than a decade in the open-source ecosystem. Similarly, Red Hat Enterprise Linux (RHEL) support for the real-time kernel has been available for several years. Still, many system administrators misinterpret its core concepts and actual operational behavior. In this article, I describe some of its key features, differences from the standard kernel, and quick installation steps.
Initially, Red Hat introduced a product called "Red Hat Enterprise MRG," where MRG stood for Messaging, Real-time, and Grid. MRG 2 was based on RHEL 5 and RHEL 6 versions, whereas MRG 3 was based entirely on RHEL 6. From RHEL 7 onward, the MRG product was deprecated, and "RHEL for Real-Time" was introduced as a separate product.
What is the real-time kernel?
A real-time kernel is not necessarily superior or better than a standard kernel. Instead, it meets different business or system requirements. It is an optimized kernel designed to maintain low latency, consistent response time, and determinism. By way of comparison, a standard RHEL kernel focuses on throughput-oriented operations and fair scheduling of tasks. The real-time kernel is also known as
The simplest way to identify a real-time kernel is to execute the
uname -r command on the terminal, and then look for the rt keyword in the kernel version. If rt is missing, then the system uses the standard kernel.
The following output is an example:
kernel-rt mechanisms include:
- A task's priority is checked (1-99) under heavy load.
- High priority (99) tasks are given preference for CPU execution.
- Does not use the Completely Fair Scheduling (CFS) policy.
- Scheduling policies in-use are SCHED_FIFO or SCHED_RR.
- Maintains low latency execution time.
- Offers an opportunity to measure, record, and configure response time.
To further understand how
kernel-rt is different from a standard kernel, the following diagram explains how using
kernel-rt maintains low latency and response times in the system.
In the above diagram, blue plots represent response time(ms) on a tuned standard RHEL7 kernel, whereas green points represent response time(ms) on a tuned RHEL7
kernel-rt. The green section shows consistency, low latency, and low response time. To study this graph and the related testing in further detail, please refer to the Red Hat official guide "Why Use Red Hat Enterprise Linux for Real Time to Optimize Latency."
Below I have listed some other differences vital to comparing RHEL for Real Time with standard RHEL:
- Kexec and vmcore - Installation and configuration are the same as standard RHEL.
- Third-party modules - All modules must be recompiled with
kernel-rtbefore being used.
irqbalancedaemon - Disabled by default.
- Scheduling policy - Only SCHED_FIFO or SCHED_RR is used.
- Interrupt (IRQs) - Uses dedicated CPUs to isolate processes from each other.
Install RHEL for real-time
Let's perform a quick installation and observe the other packages that are part of a
kernel-rt installation. A separate ISO image of
kernel-rt is available for fresh provisioning. A user/admin can also install the
kernel-rt RPM package on an existing standard RHEL system, followed by a reboot.
Step 1: Download the ISO image named RHEL for Real-Time from the Red Hat customer portal and provision. You may also use an existing standard RHEL box and enable the
rhel-7-server-rt repository to fetch
Step 2: Perform a group installation to install all RT specific packages simultaneously.
Step 3: Verify all dependent packages.
Step 4: Reboot the system, and then select
kernel-rt from the boot console.
Users should perform the RT group installation, as it provides all necessary
kernel-rt dependent packages. The following table displays brief dependent package details:
- kernel-rt - Real-time kernel
- rt-setup - Most critical package for RT environment setup.
- rtctl - Configures scheduling priorities for kernel thread.
- rt-test, rt-checks, rteval - Used to test system capabilities and loads.
Let me give you a very brief real scenario illustrating why RT group installation is crucial. One of the leading telecom customers was running real-time RHEL kernel (3.10.0-693.11.1.rt56.632.el7) as a three-blade cluster environment on HP ProLiant BL460c Gen9 machine. They observed traffic delays and timeouts in the cluster environment. A large number of UDP self-heartbeat (HB) packets were missing or dropped. We observed that there were no issues with network cards or cluster modules once we analyzed
strace packets. Further diagnosis showed that the
rt-setup package was missing because
kernel-rt was installed forcefully without any of its dependent packages.
|Missed HB's in 7 days
|Missed HB's in 7 days with rt-setup
rt-setup package is responsible for configuring and running the
rt-entsk daemon, which prevents IPI generation and delay of real-time tasks.
The following bullet points display some of the key features of
- Creates real-time groups.
- Adds real-time limits configuration for PAM.
/usr/bin/rt-setup-kdumpto configure kdump in RT.
- Adds udev-specific rules for threaded irqs and
/usr/bin/slub_cpu_partial_offto turn off
- Adds net-socket timestamp static key daemon (
It is worth the effort to run a real-time kernel and evaluate its potential as many applications can benefit from it.
Kernel-rt can be tuned to further reduce performance issues. Tools such as
ftrace are provided with RHEL for Real-Time and are very useful for analyzing and debugging latency issues. However, those details are out of the scope of this article.
The real-time kernel has been around for a long time, but many sysadmins are not sure how to work with it. In this article, I defined what the real-time kernel does. I also provided you with information on how to install it. Perhaps the real-time kernel will help your organization realize even greater Linux performance!
[ Free online course: Red Hat Enterprise Linux technical overview. ]