Crossing the threshold to dependable open source AI serving

When large open weight models first captured attention, the industry was faced with a challenge. Powerful open models were becoming available, but technologists lacked an open inference serving system that could run them efficiently. Having spent years building inference engines for CPUs, my team and I redirected all of our energy toward the vLLM project. Together with the community, vLLM capabilities expanded to become the de facto inference server. Today the vLLM project, now part of the PyTorch foundation, has a vibrant community of companies and developers actively adding support for new models, hardware, and research techniques.

As Red Hat continued this work, conversations with our enterprise users exposed a new reality. Getting a single server to perform efficiently was table stakes. The next challenge was orchestrating model serving at cluster scale. Last spring, we announced the llm-d project to address this, developing an open source, Kubernetes-native orchestration layer for distributed LLM serving across multi-tenant production clusters. Today, Red Hat and our community partners announced the release of llm-d 1.0. Reaching this milestone reflects a project hitting full maturity, shaped by hardened code, stability proven under real production load, and widespread adoption across an expanding contributor community. Open source AI serving has crossed from an experimental science project into dependable infrastructure you can build on.

What project maturity and code hardening look like on day 2

In early open source experiments, teams measure progress through isolated benchmark runs on pristine hardware. Keeping an AI service healthy on day 2, when hundreds of automated agents and human users contend for shared resources, requires an entirely different operational discipline.

Platform teams came to us with a clear dilemma: What happens when a developer debugging code on an interactive session gets stalled because an autonomous background agent flooded the GPU pool with dozens of parallel tool calls? Building for day 2 means solving these friction points directly in the request path. Over the past year, our contributor community focused on intense code hardening. We built priority-aware flow control so latency-sensitive queries from interactive users do not get starved by bursty background agents. We engineered router high availability with hot standby and coordinated rolling updates, letting operators upgrade models and runtimes without dropping live traffic. Code hardening is about stripping away operational surprises so platform teams can run distributed inference like standard enterprise plumbing rather than fragile custom code.

Stability proven under production load

Stability cannot be established on synthetic benchmarks alone. It has to survive real-world scale where agent contexts expand over hours of execution and accelerator memory faces unrelenting pressure. When multi-turn workloads repeatedly hit unmanaged clusters, naive load balancers scatter requests, wiping out cached prefix state and triggering massive recomputation costs.

llm-d delivers production-scale stability by making the orchestration layer aware of inference dynamics. It preserves context locality, disaggregates prefill from decode phases, and balances work across heterogeneous accelerator pools. In production, this operational stability creates dramatic efficiencies. At IBM Research, exploring quantum error correction hardware on llm-d generated multi-turn workloads with massive context reuse identical to agentic coding loops. That deployment yielded a 98% prefix-cache hit rate and reduced blended serving costs five to ten times compared to proprietary options.

In benchmark demonstrations, researchers used llm-d to deploy GLM-5.2, an approximately 753-billion-parameter open weight model on widely deployed NVIDIA GPUs, showcasing how enterprises can use existing GPU fleets to serve frontier-class AI at scale. On a workload with hundreds of agentic sessions running in parallel, llm-d delivered over 6.6 million output tokens per minute at peak on agentic benchmarks. The cluster sustained up to 3,000 concurrent coding agents with zero preemptions, demonstrating that llm-d can efficiently serve large mixture-of-experts models for agentic workloads.

Wide adoption across an expanding ecosystem

A mature infrastructure project earns credibility through the community that builds it and the teams that trust it when traffic spikes. Today, the llm-d community has more than 500 contributors across 25 organizations, all building openly under CNCF governance.

That shared investment is already active in production across frontier model labs, hyperscalers and neoclouds, and major enterprises. At Red Hat, we bring this hardened stack directly to our users through Red Hat AI Inference, whether they run it natively within Red Hat OpenShift AI or across third-party Kubernetes environments.

llm-d and PyTorch

Celebrating a new version of a project governed under the CNCF at PyTorch Conference North America highlights how modern computing is evolving. Distributed inference sits directly at the seam between deep learning runtimes and cloud native orchestration. What happens in the orchestration layer directly influences what model runtimes can accomplish.

Model developers and silicon partners drive rapid innovation inside PyTorch, while the CNCF community provides decades of operational rigor for distributed systems. Placing llm-d under CNCF governance was a conscious step to keep the control plane vendor-neutral and eliminate ecosystem friction. It builds an open bridge between both communities. Platform teams can orchestrate complex, multi-accelerator clusters using standard cloud native workflows, while retaining the low-level execution speed built by model authors in PyTorch.

Get involved with the llm-d and PyTorch communities

The future of open source AI will not be locked inside closed-source serving stacks. It’s being built openly by researchers, runtime developers, and platform operators working together across foundation borders. While 1.0 represents a major operational milestone, our community is already moving forward to support emerging hardware architectures, larger model shapes, and evolving agent patterns.

We invite you to help shape that future. Check out the project on GitHub, join the conversation, and collaborate with both the llm-d and PyTorch communities as we build durable infrastructure for open source AI serving.