Red Hat global preferences

Red Hat Blog

Emerging Technologies: Developments in Red Hat Enterprise Linux RealTime

You may have heard the news from Red Hat’s CTO Brian Stevens’ keynote at the Red Hat Summit that we are building a realtime variant of Red Hat Enterprise Linux. There were a few sessions at the Summit describing our initiative in more detail. For those of you who weren’t present, we’d like to share some realtime info with you here.

We can simplify the primary functional objectives of our realtime initiative down to a few key points:

  1. Determinism: provide consistent, repeatable response times
  2. Priority: ensure the highest priority processes run first

Sounds rather basic, right?

For most workloads, a properly tuned Red Hat Enterprise Linux kernel (in RHEL 2.1, 3, 4 or 5) meets customer requirements. Typical timing requirements are in the range of millisecond response time. However for the most demanding customer workloads, the requirements are in the microsecond range. To give just a few representative examples:

  • Financial Services industry: here time is money. In this highly competitive market, shaving fractions of a second off the time it takes to performs market analytics yields huge advantage. Additionally, there are increasing government regulations for consistency in trading. They don’t want things smelling fishy if some trades take longer than others.
  • Federal command and control systems: here, “close enough” isn’t good enough. They need to dependably know that the highest priority application threads will run and complete in predictable periods of time.

The primary reason why the standard Red Hat Enterprise Linux products can’t completely meet the most demanding response time requirements is because there are numerous lengthy kernel codepaths which are non-preemptable. Without getting too technical here, this means that while these non-preemptable kernel codepaths are running, the high priority application threads are not running. Hence these long-running kernel codepaths result in delays in the application running, which is the cause of inconsistent response times (also referred to as non-determinism). In his keynote address at the Summit, Brian displayed a performance chart illustrating that when running a messaging workload on standard Red Hat Enterprise Linux, there was substantial deviation in response times. Whereas when running the realtime kernel, the response times were highly consistent.

Note, of course, that realtime is not going to be appropriate for every customer. Hence our plan to continue with the regular Red Hat Enterprise Linux kernel while also providing a realtime kernel. Breaking up existing non-preemptive kernel codepaths incurs a performance cost that is unlikely to be warranted for the majority of customers. The standard Red Hat Enterprise Linux kernels focus on providing leadership throughput. But the cost of this throughput is sometimes a specific application may experience a delay (scheduling, I/O, etc). For realtime, the goal is for applications to experience a consistent (deterministic) level of performance (meaning latency) 100 percent of the time. For non-realtime the goal is for the complete system to deliver the maximum performance (meaning throughput), with the occasional application delay being acceptable. Hence, it’s a workload dependent tradeoff.

Over two years ago, a Red Hat engineer named Ingo Molnar began his journey to integrate realtime capabilities in the mainline upstream Linux kernel. While other companies had tried and failed in getting highly invasive realtime kernel capabilities included in the past; Ingo took a fundamentally different approach. Ingo took a slow, methodical approach of incrementally developing a progression of performance enhancements that proved to be beneficial to general purpose workloads. Inspired by Ingo’s initial success, a growing community of kernel development has rallied around what is now called the “-rt” project.

Today, Ingo continues to lead the mainline Linux kernel realtime initiative. The -rt project includes additional Red Hat kernel hackers as well as participants from other companies such as IBM. Other companies in the traditional embedded space have been participating in porting to architectures other than x86/x86-64 – certainly beneficial as architectural diversity always improves the overall robustness. This collective band of engineers has worked closely together over the past year resulting in a huge set of realtime kernel enablers being incorporated into mainline. In fact, many of the recent record-breaking performance benchmarks recently announced on Red Hat Enterprise Linux 5 owe credit to this crew. This is a great example of our philosophy of open and inclusive community development. Hats off to the many Red Hat kernel engineers, as well as talented developers at other companies too.

While Red Hat Enterprise Linux 5 does have many realtime-related enhancements, there remains a large set of additional features to be developed before we can meet the most demanding customer requirements for determinism. Our work is not done. To satisfy this strong customer demand for enhanced realtime capabilities, Red Hat has established a productization initiative. The model is similar to the proven strategy we use for Red Hat Enterprise Linux. That is, we continue to progress our major development efforts in the mainline community trees. From that we pick a stabilization point and branch the product line – where hardening, testing (both internal and external with customers and partners) and ongoing support maintenance occurs.

At this time we have test versions of a realtime variant of Red Hat Enterprise Linux 5 available for validation by our most demanding customers (a tough crowd – but we love them dearly). Once we get the thumbs-up from testing in real-world scenarios, we will be able to share more details on the product timeframe and cost.

Don’t tell Ingo this, but there’s more to Red Hat Enterprise Linux than just the kernel. In the course of product development of a realtime variant of Red Hat Enterprise Linux, we are taking a broader view of customer deployment requirements. For example, IBM is a co-development partner with our realtime team to offer the only realtime-certified Java runtime on Linux. Additionally, we recognize that enterprise realtime deployments often have requirements for high speed messaging. This need will be met by the highly complementary AMQP (Advanced Message Queue Protocol) high-speed messaging middleware project that Red Hat is spearheading. Red Hat’s AMQP messaging middleware will be optimized for the realtime variant of Red Hat Enterprise Linux.

As you can see, there is a lot going on here at Red Hat in support of realtime capabilities. Developers (both from Red Hat and external companies) are actively cooperating in their respective upstream communities. We’ll keep you posted as product plans solidify. Meanwhile, if you think that your company’s workload has stringent determinism requirements, consider contacting your Red Hat rep to become part of the upcoming realtime beta testing.