llm-d: Breaking the cost and capacity barriers

The AI industry is shifting its attention from training models to running models more efficiently. Enterprise AI applications generate millions of inference requests as they coordinate multiple models, tools, and agents. Inference is the process where a trained AI model generates a response to a user prompt. Every request consumes computing capacity, making inference efficiency one of the primary drivers of both AI performance and infrastructure cost.

The challenge is both acquiring enough capacity and using that capacity intelligently. As model parameters have grown exponentially in size and complexity, traditional deployment approaches have struggled to keep up. Many organizations still rely on standard Kubernetes load balancing to distribute inference requests,  which was originally designed for stateless applications. 

However, large language model (LLM) inference behaves differently. LLMs depend heavily on the key-value (KV) cache, the model's short-term memory, to avoid repeating work. When conventional load balancers blindly distribute related requests evenly across different inference servers, cache locality is lost. The result is redundant computation, unnecessary GPU utilization, and unnecessary infrastructure costs as the model is forced to recompute work it has already performed.

From better models to better inference

Scaling AI isn't simply a matter of adding more inference servers, inference infrastructure needs to be LLM-aware. Open source has played a central role in making AI models more accessible, and 2 open source projects, vLLM and llm-d, are doing the same for inference.

• vLLM is an open source inference engine optimized for running models more efficiently on a single node or server.
• llm-d acts as the overarching control plane for distributed inference. While vLLM increases single-node efficiency, llm-d spans multiple vLLM instances to orchestrate and optimize the entire cluster.

Using intelligent, KV cache-aware scheduling, llm-d understands where computed prompts already exist and routes inference requests accordingly. By preserving cache locality across multiple inference instances, it minimizes redundant computation, improves GPU utilization, reduces latency, and helps lower the cost of serving AI workloads. Capabilities such as intelligent request routing and inference disaggregation also help organizations to extract more value from the infrastructure they already have instead of simply adding more hardware.

Looking ahead

Open source projects like vLLM and llm-d demonstrate how collaborative innovation is helping organizations overcome the cost and compute barriers that stand between AI experimentation and enterprise production at scale.

In Part II of this series, we'll explore why efficient inference is only one piece of the puzzle, and why infrastructure independence is that path to sovereign AI. 

To learn more, watch Red Hat's Brian Stevens and Robert Shaw discuss the fundamental challenges of scaling AI inference.

• Watch the full discussion
• Catch a quick highlight
• Read the blog post: What is llm-d and why do we need it?
• Learn more about llm-d