Using Podman on RHEL for Real Time

A real-time operating system (RTOS) is like a skilled orchestra conductor. A conductor coordinates and manages the timing and synchronization of many musicians and instruments to create a harmonious performance. An RTOS orchestrates the timing and execution of multiple tasks and processes in a computer system so that critical operations are completed precisely on time. This is known as determinism, and it's the most important feature of RTOS because responses to inputs need to be returned within a set maximum time frame, usually measured in micro (µ) seconds.

If you're using or looking into an RTOS for a project that demands deterministic results, where there is an absolute limit on response time, you might find some vendors of hard RTOS to be expensive. Hard RTOS is used in avionics, nuclear control systems, and some defense applications. Alternatively, a soft RTOS fulfills a requirement to respond deterministically, with the quickest responses tied directly to the level of tuning performed with hardware and software.

Some of the use cases for using a soft RTOS include high-frequency trading in financial markets, distributed simulations, telecommunications, and C3 (command, control, and communications). Soft RTOS are used in the financial, telecommunications, and defense industries. Determinism is a requirement for these industries. Slow trades on Wall Street can mean millions of dollars lost. A telecommunications system that takes too long to move packets results in poor call quality.

Red Hat Enterprise Linux for Real Time

Soft RTOS, with proper tuning, can meet mission requirements and enable your team to develop on an enterprise open source Linux platform. Red Hat Enterprise Linux for Real Time (RHEL for Real Time) is a soft RTOS that provides deterministic results with access to all of the standard userspace tooling for faster RTOS application development and system management.

Included with this is Podman, an open source, daemonless, Linux-native Open Container Initiative (OCI) container tool that helps you find, run, and manage your containers. Podman optimizes efficiency for edge and tactical deployments that require a deterministic response.

The RHEL for Real Time kernel is based on kernel-rt, and provides deterministic performance, reduced latency, improved system response, resource isolation, and compatibility with Real Time tools. RHEL for Real Time is about being deterministic, on time, in a predictable way.

For the Department of Defense, an example use case would be an edge location, processing incoming intelligence data used for Multi Domain Operations. Intelligence data has a shelf life that demands the ability to process and publish data in a determined amount of time.

The graph below, from the Open Source Automation Development Lab, compares a million samples of machines that use the RHEL 9 and the RHEL for Real Time kernel.

• The blue points in this graph represent the system response time (in microseconds) of machines running a tuned RHEL 9 kernel.
• The green points in the graph represent the system response time of machines running a tuned real-time kernel. 

It's clear from this graph that the response time of the Real Time kernel is consistent, in contrast to the variability of the standard kernel.

Real Time Kernel

To optimize performance, the tuning of both your hardware and RHEL must occur before the installation of a real-time kernel. Without tuning, you will see little performance improvement and a wide range of response times.

To get started with containers on RHEL for Real Time, we will switch to the root user to install and run everything for this example.

To begin, install the container-tools package group:

$ sudo dnf install container-tools

Next, subscribe to the RHEL for Real Time repository, and then install the Real Time kernel and associated packages:

$ sudo subscription-manager repos --enable rhel-9-for-x86_64-rt-rpms
$ sudo dnf groupinstall RT

Once the install is complete, reboot the machine so that the Real Time kernel is active. Verify that you see rt in the kernel version, and make sure it's selected.

Make this your default kernel:

$ ls /boot/vmlinuz* | grep rt
/boot/vmlinuz-5.14.0-284.30.1.rt14.315.el9_2.x86_64
$ sudo grubby \ 
--set-default=/boot/vmlinuz-5.14.0-284.30.1.rt14.315.el9_2.x86_64

Create a directory to run your container test in:

$ mkdir rtcontainer
$ cd rtcontainer

Log into a host that provides a container registry service using Podman:

$ podman login registry.redhat.io
Username: my_customer_portal_login
Password:
Login Succeeded!

Create the following Docker file:

$ vim Dockerfile
FROM rhel8
RUN dnf -y install rt-tests
ENTRYPOINT cyclictest --smp -p95

Build the container image from the directory containing the Dockerfile:

$ podman build -t cyclictest .

Once you run this, view your container image:

$ podman images
REPOSITORY           TAG        IMAGE ID    CREATED   SIZE
localhost/cyclictest latest     73dc69...   13 sec    484 MB
registry.access.redhat.com/ubi8 27e761...   13 days   215 MB

Now run the container:

$ podman run --device=/dev/cpu_dma_latency \
--cap-add ipc_lock --cap-add sys_nice \
--cap-add sys_rawio --rm -ti cyclictest

The first-in first-out (FIFO) scheduler policy is made available for workloads running inside the container through the --cap-add=sys_nice option. This option also allows setting the CPU affinity of threads, another important configuration dimension when tuning a real time workload.

The --device=/dev/cpu_dma_latency option makes the host device available inside the container (subsequently used by the cyclictest workload to configure the CPU idle time management). If the specified device is not made available, an error similar to the message below appears:

WARN: stat /dev/cpu_dma_latency failed: No such file or directory

If the container is built and run as a non-privileged user, then you get the following warning:

WARN: stat /dev/cpu_dma_latency failed: Permission denied 

Running this cyclictest provides you with the maximum detected latency, which would be the worst case latency length on the evaluated system.

$ podman run --device=/dev/cpu_dma_latency \
--cap-add ipc_lock --cap-add sys_nice \
--cap-add sys_rawio --rm -ti cyclictest
# /dev/cpu_dma_latency set to 0us
policy: fifo load avg: 2.33 1.95 1.73 1/567 3
T:0 (2) P:95 I:1000  C:194224 Min:3 ACT:162 Avg:16 Max:3154
T:1 (3) P: 95 I:1500 C:129513 Min:3 ACT: 68 Avg:29 Max:5523

The system I am running has not been tuned. You can see in the Max: field that my maximum latency was 5523 microseconds.

You can use tuna to tune your system and optimize performance while running applications, and hwlatdetect and rteval to get latency reports.

RHEL as your RTOS

There are many advantages to Red Hat when looking for a RTOS. You have access to container management tools like Podman, Red Hat’s huge ecosystem of certified software and hardware partners, a long life cycle for each major RHEL release, and an award-winning support website. RHEL for Real Time offers determinism, reduced latency, improved system responsiveness, and helps accelerate developer productivity. These advantages make for a powerful combination when deploying real-time containerized applications.