I was invited to do a presentation at the High Performance on Wall Street conference in New York City on September 17 to discuss Red Hat’s realtime development project. For the benefit of those not in attendence, I’ll briefly recap the main points of my talk here.

My main intention was to help the audience become more informed. I consider this important because realtime offerings have been getting a lot of buzz lately, but the term itself and true benefits of realtime are often not well understood. Realtime is a great fit for many deployments, but it is not a silver bullet for all performance woes. The more informed the customer is, the better prepared they are to follow the right path to their performance objectives.

The main points to be explored:

  1. Red Hat Enterprise Linux today provides industry leadership performance on a diversity of industry-standard benchmarks
  2. Defining realtime in the broader picture
  3. System tuning is a crucial step in achieving high performance
  4. Highlights of new realtime kernel features
  5. Highlights of new high-performance middleware

Keep reading for more details on each.

Red Hat Enterprise Linux today provides industry leadership performance on a diversity of industry-standard benchmarks

There are a number of industry-leadership performance results that provide us with a variety of benchmarks for Red Hat Enterprise Linux performance. Some include:

  • World record TPC-H performance @300GB with Red Hat Enterprise Linux/HP/Oracle. See here.
  • Hot-off-the-press, world record SPECweb2005 on Intel quad core on HP. See here.
  • New record performance results from the Securities Technology Analysis Center running RMDS (Reuters Market Data System) on Intel quad core. See here.

The point here is that we’ve established a leadership baseline.

Defining realtime in the broader picture

The term “realtime” can mean many things. For Red Hat, we equate realtime with determinism – meaning consistent, predictable results. For many customer-work cases, the most important thing is to be able to depend on the completion of a computation in a predictable period of time. Some representative cases include:

  • Financial-trading applications: where if it takes too long to complete a trade or financial model calculation, a lost trade results.
  • Defense contractors: including the work we are doing with IBM and Raytheon on the realtime capabilities in an upcoming Naval warship.
  • Telecommunications: network switch providers needing to respond to network traffic in small, finite time windows

Typically, realtime discussions only refer to the kernel. Through our discussions with customers, it has become clear to us that there are broader requirements. For example, in the use cases we are targeting, computers are typically networked together – where the response time is typically round-trip among systems. Additionally, many target deployments involve requirements for realtime Java. Hence Red Hat’s realtime initiative takes a broader context in the form of:

  • A new realtime kernel
  • A high-speed messaging middleware offering
  • Close partnership and co-development with IBM to certify their WebSphere realtime java on this solution stack

System tuning is a crucial step in achieving high performance



Improved determinism of realtime kernel.

This performance graph plots results of response times for a client/server Tibco EMS messaging workload. This primarily measures round-trip messaging response time. The graph shows that when the workload was run on standard Red Hat Enterprise Linux 5, straight out of the box, with no tuning, that the results varied considerably. From a low of about 1.5 ms (milliseconds) to over 150 ms – a range of over 10x. As mentioned above, the goal of realtime software is to provide deterministic behavior. Clearly a range of 10x isn’t all that deterministic.

This same graph has another plot of the same messaging benchmark running on the same exact software, except in this case the system was tuned. This substantially improved the results, delivering a response time range of 1.400 ms to about 1.575 ms. That is a HUGE difference from a determinism perspective, changing the range from 10x down to around 10 percent.

The third plot on this same graph depicted our upcoming realtime kernel running on the same hardware and Red Hat Enterprise Linux 5 distribution. The realtime results illustrated a consistent performance in the 1.455 ms to 1.475 ms range.

For the majority of customers, the level of determinism possible via proper system tuning on standard Red Hat Enterprise Linux 4 or Red Hat Enterprise Linux 5 meets their requirements. However for the most demanding set of customers, the improvements in determinism offered via our realtime kernel is an important competetive advantage.

From an informed consumer perspective, the following is important:

  • Proper tuning of standard Red Hat Enterprise Linux yields a level of determinism which will satisfy the majority of customers.
  • Realtime kernels (from any vendor) are not a panacea – meaning that until you have performed the system tuning which provides the vast majority of the determinism improvements you will be unlikely to see the benefits of a realtime kernel. Hence its not as simple as just dropping in a new kerenl and magically all performance woes disappear. In fact, be very leary of any realtime marketing collateral which suggests such easy magic.
  • Only after you’ve maxed out the system’s determinism through system tuning should you elect to use a realtime offering to get that last 10 percent or so.

Highlights on new realtime kernel features

Red Hat has been leading the upstream Linux community in the advancement of realtime kernel capabilities. The primary objective of these changes is to eliminate long-running, non-preemptable kernel codepaths. Basically, removing bottlenecks which can stall the running of high-priority application tasks. In so doing, the overall observed determinism increases as we eliminate the sources of such stalls.

Our historical experience with Red Hat Enterprise Linux convinced us that the most productive way to develop new features is to do so in a completely open and inclusive manner. We are applying the same proven approach to realtime development.

We’ll have a whitepaper topic on this coming soon.

Highlights on new high-performance middleware

Of course, realtime determinism is not the only performance-related technology required in the financial services market. Another critical software component is messaging middleware. Historically, this has been the exclusive province of proprietary offerings, resulting in the usual proprietary vendor lock-in and interoperability issues.

Over the past year there has been an intensive cooperative development initiative involving Red Hat, several financial institutions and network switch providers to develop both specification and reference implementations for messaging middleware. The result is referred to as AMQP (Advanced Message Queuing Protocol).

The definition of an open specification has enabled the development of multiple interoperable implementations. For example, Red Hat has one implementation that it is developing as an open source project, while other companies are creating proprietary implementations. These all adhere to the specification in an interoperable manner across a variety of programming languages – including Microsoft’s .net framework.

AMQP capabilities include a flexible producer/consumer exchange model. This facilitates reliable, transaction-based messaging in a variety of configurations from point-to-point to multicast style.

And Now?

Expect more information soon from Red Hat detailing these messaging capabilities. The combination of realtime kernel, AMQP messaging and realtime java (via our partnership with IBM) will become available soon.

See the full slide deck and listen to an mp3 of this presentation here.