Accelerate AI inference with vLLM

At this point, the transformative potential of a large language model (LLM) is clear, but efficiently deploying these powerful models in production can be challenging.

This challenge is not new. In a recent episode of the Technically Speaking podcast, Chris Wright spoke with Nick Hill, a principal software engineer at Red Hat who worked on the commercialization of the original IBM Watson "Jeopardy!" system years ago. Hill noted that these early efforts focused on optimizing Watson down from a room full of servers to a single machine, establishing that systems-level engineering is key to making powerful AI practical.

Wright and Hill also discussed how this same principle applies to modern LLMs and the vLLM open source project, which is revolutionizing AI inference by making AI more practical and performant at scale.

What is vLLM?

vLLM is an inference server that directly addresses the efficiency and scalability challenges faced when working with generative AI (gen AI). By maximizing the use of expensive GPU resources, vLLM makes powerful AI more accessible and practical.

Red Hat is deeply involved in the vLLM project as a significant commercial contributor. We have integrated a hardened, supported, and enterprise-ready version of vLLM into Red Hat AI Inference Server. This product is available as a standalone containerized offering, or as a key component of the larger Red Hat AI portfolio, including Red Hat Enterprise Linux AI (RHEL AI) and Red Hat OpenShift AI. Our collaboration with the vLLM community is a key component of our larger open source AI strategy.

Why vLLM matters for LLM inference

LLM inference is the process in which an AI model applies its training to new data or queries, and it has some inherent bottlenecks. Traditional inference methods can be inefficient due to sequential token generation and low GPU utilization, leading to high latency under load, inflexible architectures that are unable to scale, and constraints on memory bandwidth.

vLLM offers a streamlined approach. Its primary goal is to maximize GPU utilization and throughput, and it achieves this through a series of key optimizations.

• PagedAttention: This core innovation uses a concept similar to a computer's virtual memory to efficiently manage the key-value (KV) cache. The KV cache is the intermediate data a model needs to remember from one token to the next.
• Continuous batching: This technique allows the inference server to efficiently process new incoming requests while a batch is already being processed, reducing idle time and increasing overall throughput.
• Other critical optimizations: vLLM also leverages techniques like speculative decoding, which uses a smaller, faster model to predict the next tokens, and optimized CUDA kernels to maximize performance on specific hardware.

vLLM acts as an interface layer that helps manage the overall data flow, batching, and scheduling, enabling LLMs to integrate with a wide array of hardware and applications.

Strategic advantages for enterprise AI

While vLLM is technically interesting, it also provides important strategic benefits for IT leaders. vLLM's optimizations can help you manage costs, scale more effectively, and maintain tighter control over your technology stack.

Democratizing AI and optimizing costs

vLLM helps your organization get more out of its existing hardware. By significantly increasing GPU utilization, it helps reduce the amount of hardware needed to run your workloads, which in turn helps reduce costs. This makes advanced AI capabilities more attainable for more organizations.

Scaling AI applications with confidence

The enhanced GPU utilization and faster response times translate directly to supporting larger model and application deployments. Your organization can serve more users and handle more complex AI workloads without compromising performance. This helps provide the enterprise-grade scalability that is essential for moving AI projects from a proof of concept to a production environment.

Hardware flexibility and expanding choice

The open source nature of vLLM and its broad support for various hardware accelerators from companies like NVIDIA, AMD, and Intel—along with leading models from providers like Meta, Mistral, and IBM—is a key strategic advantage. This gives your organization more flexibility when selecting hardware solutions and helps you maintain the ability to choose accelerators that work best for your unique needs, even if they’re dynamic.

Accelerated innovation and community impact

The value of vLLM's active open source community is substantial. The community is active and growing, which leads to rapid integrations of new research and advancements. This fast-paced development and innovation has helped establish vLLM as a standard for LLM inference, and your enterprise can continuously benefit from the latest innovations.

Enterprise-grade AI with vLLM

Red Hat's vision is to make AI practical, transparent, and accessible across the hybrid cloud. vLLM is a cornerstone of this strategy, and a key factor in our guiding vision, "any model, any accelerator, any cloud."

Red Hat AI Inference Server

We have integrated vLLM into Red Hat AI Inference Server—a hardened, supported, and enterprise-ready distribution of vLLM. In addition to our repository of optimized and validated third-party models, we provide  tools like LLM Compressor, which helps deliver faster and more cost-effective deployments across your hybrid cloud environments.

Just as Red Hat helped unify the fragmented Linux landscape, the Red Hat AI Inference Server, powered by vLLM, provides a similar unifying layer for AI inference. This helps simplify complex deployments for organizations that need a consistent and reliable way to run AI workloads.

Unifying the AI infrastructure

Red Hat AI Inference Server is available as a standalone containerized offering. It also plays an integral role across the Red Hat AI portfolio:

• The core components are included with Red Hat Enterprise Linux AI (RHEL AI), which provides a foundational platform for LLM development, testing, and deployment.
• It is a key component within Red Hat OpenShift AI, an integrated MLOps platform for managing the full lifecycle of AI models at scale.
• Additionally, our Hugging Face repository of optimized models offers access to validated third-party models that are pre-optimized to run efficiently on vLLM, such as Llama, Mistral, Qwen, and Granite.

Our commitment to the open source community is ongoing. In addition to our involvement with the vLLM community, we also recently launched the llm-d project, a Kubernetes-native, high-performance distributed LLM inference framework that incorporates vLLM. This new initiative includes other contributors like Google and NVIDIA, and is designed to help run gen AI at a massive scale, helping deliver competitive performance for most models across various hardware accelerators.

How Red Hat can help

Red Hat AI provides a complete enterprise AI platform for model training and inference that delivers increased efficiency, a simplified experience, and the flexibility to deploy anywhere across a hybrid cloud environment. Our vision is to make AI practical, transparent, and accessible, and our portfolio is designed to help you build and run AI solutions that work for your business, from initial experiments to full production.

Our hybrid cloud approach gives you the freedom to implement AI in any way you choose, whether you need to modernize existing applications or build new ones. We also offer AI training and certification, including no-cost AI Foundations courses, to help your teams develop the AI skills your organization so sorely needs.