Red Hat and NVIDIA: Setting standards for high-performance AI inference

Red Hat is proud to announce industry-leading results from the latest MLPerf Inference v6.0 benchmarks, achieved through deep engineering co-design with NVIDIA. These results demonstrate that when you combine Red Hat’s open-source leadership with NVIDIA’s leading AI infrastructure, the result is a versatile, proven platform ready for any enterprise inference workload—from vision and speech to complex reasoning.

Our latest submissions focused on maximizing the potential of the NVIDIA HGX H200 and NVIDIA HGX B200 systems, proving that software optimization is just as critical as raw horsepower for achieving peak ROI.

Results at a glance

Across language, vision, and speech models, Red Hat’s stack delivered top-tier throughput and latency results on NVIDIA AI infrastructure.

Qwen3-VL-235B (multimodal vision model)

The Qwen3-VL-235B model, a massive 235-billion-parameter multimodal vision-language model, represents a significant challenge for inference engines due to highly variable image resolutions. Using NVIDIA Blackwell GPUs running on Red Hat Enterprise Linux (RHEL) with vLLM and NVIDIA Dynamo, we achieved the highest offline throughput in our class. Notably, our Blackwell submission exceeded the next top performer by 50% in the Server scenario.

Key engineering wins:

• Triton-based improvements: Optimizations to the vision encoder yielded 30-40% faster ViT processing.
• FlashInfer Mixture-of-Experts (MoE) kernels: These specialized kernels handled the MoE architecture with extreme efficiency.
• FP8 Multimodal Attention: Leveraging NVIDIA’s advanced data formats to lower cost per token without sacrificing accuracy.

GPT-OSS-120B

Our submission for GPT-OSS-120B marks the first time a model of this scale has been benchmarked on Kubernetes infrastructure for MLPerf. By using Red Hat OpenShift AI and the llm-d scheduler, we demonstrated that distributed inference can scale effectively on NVIDIA AI infrastructure (H200 and B200 GPUs) while maintaining strict latency requirements.

We adopted a two-pronged strategy to optimize inference performance. First, our Bayesian optimization–based hyperparameter tuning pipeline on OpenShift identified an optimal configuration for a single replica that reduced P99 time-to-first-token (TTFT) from 3.4 seconds to 2.1 seconds (~38% improvement), meeting the sub-3s target.

Second, we optimized multi-replica performance by refining our load balancing and scoring strategy. By analyzing request distribution across replicas, we improved utilization and minimized tail latency, enabling more consistent scaling under load.

Whisper large-V3 (speech-to-text)

We submitted Whisper-large-v3 results on NVIDIA H200 and NVIDIA L40S GPUs, both running Red Hat Enterprise Linux (RHEL) and vLLM.

• 8x H200 offline: 36,396 tokens per second, the leading H200 result, 13% faster than the next closest submission                                            
• 2x L40S offline: 3,647 tokens per second, the first and only L40S submission for Whisper in MLPerf Inference v6.0

These results were driven by a systematic ablation study across config parameters to identify the optimizations that matter most for Whisper inference. Batch size tuning delivered a 40% throughput gain by maximizing GPU utilization, asynchronous scheduling contributed a further 12.8% by eliminating CPU-GPU synchronization stalls, and CUDA Graphs provided an additional 6%. With L40S widely deployed in cost-sensitive environments, our results show that an open-source inference stack delivers world-class speech recognition performance across both high-end and cost-efficient hardware.

Delivering greater efficiency and ROI

Red Hat’s software stack utilizes NVIDIA inference software Dynamo and Red Hat AI’s vLLM and llm-d to deliver significant efficiency gains on NVIDIA accelerated computing infrastructure. By optimizing every layer of the stack—from the RHEL kernel to the inference engines—we help enterprises lower their cost per token and improve overall ROI on their NVIDIA investments. Whether you are deploying on-premises or in the cloud, Red Hat provides a proven, high-performance foundation for the next generation of agentic and multimodal AI.

Want to replicate our results? Here’s how… Repo

Check out the full MLPerf Inference v6.0 results at mlcommons.org and learn more about Red Hat AI.