Listen to an automated reading of this article. * [Topics](/en/topics "Topics") * [Artificial intelligence](/en/topics/ai "Artificial intelligence") * What is vLLM? What is vLLM? ============= Updated May 12, 2026•*6*-minute read Copy URL Jump to section --------------- What is vLLM?How does vLLM work?vLLM and PagedAttentionHow can vLLM help your organization?Why vLLM is becoming a standard for enhancing LLM performanceThe vLLM communityHow Red Hat can help What is vLLM? ------------- vLLM, which stands for virtual large language model, is a library of open source code maintained by the [vLLM community](https://github.com/vllm-project). It helps [large language models (LLMs)](/en/topics/ai/what-are-large-language-models) perform calculations more efficiently and at scale. vLLM includes both an [inference](/en/topics/ai/what-is-ai-inference) server (which manages network traffic), and an inference engine (to maximize computational speed). It works by speeding up the output of [generative AI](/en/topics/ai/what-is-generative-ai) applications by making better use of the GPU memory through it’s PagedAttention algorithm. The overall goal of vLLM is to maximize throughput (tokens processed per second) to serve many users at once. [Explore Red Hat AI](/en/products/ai) How does vLLM work? ------------------- To understand the value of vLLM, it’s important to understand what an inference server does, and the baseline mechanics of how an LLM operates. From there, we can better understand how vLLM plays a role in improving the performance of existing language models. [Catch up on vLLM Office Hours for the latest developments](/en/events/ai) ### What is an inference server? An inference server is software that helps an AI model make new conclusions based on its prior training. [Inference](/en/topics/ai/what-is-ai-inference) servers feed the input requests through a [machine learning](/en/topics/ai/what-is-machine-learning) model and return an output. [Why you should care about inference](/en/artificial-intelligence/inference) To infer is to conclude based on evidence. You may see your friend’s living room light on, but you don’t see them. You may infer that they’re home, but you don’t have absolute evidence to prove it. A language model also doesn’t have absolute evidence about the meaning of a word or phrase (it’s a piece of software), so it uses its training as evidence. In a series of calculations based on data, it generates a conclusion. Just like when you calculate that if the light is off, it means your friend is out. [Find benefits and use cases of fast, efficient inference](/en/products/ai/fast-efficient-inference) ### LLMs use math to make conclusions When an LLM is being trained, it learns via mathematical calculations. When an LLM is generating a response (inference), it does so by performing a series of probability calculations (more math). In order for an LLM to understand what you’re requesting from it, the model needs to understand how words relate to each other and how to make associations between words. Instead of learning about semantics and reasoning with words like humans do, LLMs “reason” with…you guessed it, math. When an LLM responds to millions of users per day, it’s doing a lot of calculations. Processing all these calculations at once while in an application is live can be challenging. This is because (traditionally) the processing power involved in running an LLM can quickly consume significant memory. vLLM architecture upgrades continue to improve resource efficiency for things like memory and speed. [What does AI look like at the enterprise?](/en/topics/ai/what-is-enterprise-ai) Red Hat AI ---------- [Learn more about this product](/en/products/ai "Learn more about this product") vLLM uses PagedAttention to process calculations more efficiently ----------------------------------------------------------------- A landmark study, [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180), identified how existing LLM memory management systems do not organize calculations in the most efficient way. PagedAttention is a memory management technique introduced by vLLM that’s inspired by virtual memory and paging systems within operating systems. This research recognizes how the key value (KV) cache (short term memory of an LLM) shrinks and grows during throughput, and offers vLLM as a solution for managing space and computing power in a more stable way. Essentially, vLLM works as a set of instructions that encourage the KV cache to create shortcuts by continuously “batching” user responses. Before we move forward, let’s quickly define what KV cache and continuous batching are. ### What is KV cache? KV stands for key value. Key value refers to the way that an LLM formulates a meaning for a word or phrase. Let’s say you’re processing the key value for an item on a menu: French fries (key) is charged at $3.99 (value). So, when a cashier rings up an order of french fries, the computed “value” of that “key” is $3.99. LLMs process KVs similarly in that they hold the corresponding value to each key (or token) in their cache. Cache refers to a short term memory storage. Think about your personal computer–when things are running slowly, it’s common practice to “clear your cache” to make room for better, faster processing. ### What is continuous batching? Continuous batching is a technique used to process multiple queries simultaneously, with the aim of improving overall processing efficiency. Consider this: A chatbot is getting thousands of queries each minute, and many of those queries pose similar questions, like, “what is the capital of India?” and “what is the capital of Ireland?” Both of these queries include the words “what is the capital of”, which is a string of tokens (words) that the LLM has to do a lot of calculations to create meaning from. With vLLM, the chatbot can hold this string of tokens (“what is the capital of”) in a short term memory (KV cache) and send a single “translation request” rather than 2 separate ones. In other words, instead of generating a brand new response, vLLMs allow the KV cache to hold memory and create shortcuts for new queries that are similar to previously computed calculations. Processing the calculations of similar queries in a batch (rather than individually) improves throughput and optimizes memory allocation. vLLM can help [optimize memory and expand token capacity](https://neuralmagic.com/blog/enhancing-deepseek-models-with-mla-and-fp8-optimizations-in-vllm/) for larger batch sizes and long-form context tasks. [What is Model-as-a-Service?](/en/topics/ai/what-is-models-as-a-service) How can vLLM help your organization? ------------------------------------ vLLM allows organizations to “do more with less” in a market where the hardware needed for LLM-based applications comes with a hefty price tag. Building cost-efficient and reliable LLM services requires significant computing power, energy resources, and specialized [operational skills](/en/products/ai/operationalize-ai). These challenges effectively put the benefits of customized, deployment-ready, and more [security-conscious AI](/en/topics/ai/what-is-ai-security) out of reach for many organizations. It also creates major hurdles for entities seeking to build a [sovereign AI](/en/topics/ai/sovereign-ai) system. vLLM and [PagedAttention](https://arxiv.org/abs/2309.06180), the algorithm it’s built on, aim to address these challenges by making more efficient use of the hardware needed to support AI workloads. ### Benefits of vLLM Using vLLM as an inference server for LLMs has benefits such as: **Faster response time:** By some calculations, vLLM achieves up to [24x higher throughput](https://blog.vllm.ai/2023/06/20/vllm.html) (how much data an LLM can process) compared to Hugging Face Transformers, a popular open source library for working with LLMs. **Reduced hardware costs:** More efficient use of resources means fewer GPUs are needed to handle the processing of LLMs. For organizations working with extra large LLMs (those with hundreds of billions of parameters), vLLM can help maintain efficiency. Specifically, vLLM can be used alongside techniques like [distributed inference](/en/topics/ai/what-is-distributed-inference) to make the most of existing hardware and cut down on costs. **Scalability:** vLLMs organize virtual memory so the GPU can handle more simultaneous requests from users. This is especially important for [agentic AI](/en/topics/ai/what-is-agentic-ai) applications, which must process many simultaneous requests in order to complete a single multi-step task. **Data privacy:** Self-hosting an LLM with vLLM provides you with more control over data privacy and usage compared to using a third-party LLM service or application like ChatGPT. **Open source innovation:** Community involvement in maintaining and sustaining vLLM allows for consistent improvements to code. Transparency in how users can access and modify code provides freedom for developers to use vLLM in whatever way meets their needs. [Find out how to deploy Llama with vLLM](https://neuralmagic.com/blog/deploy-llama-3-8b-with-vllm/) Why vLLM is becoming a standard for enhancing LLM performance ------------------------------------------------------------- vLLM is for developers who need to squeeze every drop of performance potential from their servers and deploy reliable, scalable applications. For teams moving toward large-scale production, vLLM provides the foundation needed to reliably and efficiently serve LLMs at scale. [vLLM vs. Ollama: When to use each framework](https://www.redhat.com/en/topics/ai/vllm-vs-ollama) PagedAttention is the primary algorithm that came out of vLLM. However, PagedAttention is not the only capability that vLLM provides. Additional performance optimizations that vLLM can offer include: * PyTorch Compile/CUDA Graph - for optimizing GPU memory. * Quantization - for reducing memory space required to run models. * Tensor parallelism - for breaking up the work of processing among multiple GPUs. * Speculative decoding - for speeding up text generation by using a smaller model to predict tokens and a larger model to validate that prediction. * Flash Attention - for improving the efficiency of transformer models. Aside from the optimization abilities vLLM offers, its flexible nature has also helped it grow in popularity. vLLM works with both [small and large language models](/en/topics/ai/llm-vs-slm) and integrates with popular models and frameworks. Finally, the open-source nature of vLLM allows for code transparency, customization, and faster bug fixes. [Learn how vLLM supports inference for the open source community](https://neuralmagic.com/blog/vllm-brings-fp8-inference-to-the-open-source-community/) ### vLLM and llm-d [llm-d](/en/topics/ai/what-is-llm-d) is an open source framework that integrates and builds on the power of vLLM. It's a recipe for performing [distributed inference](/en/topics/ai/what-is-distributed-inference) and was built to support increasing resource demands of LLMs. Think of it this way, if vLLM helps with speed, llm-d helps with coordination. vLLM and llm-d work together to intelligently route traffic through the model and make processing happen as quickly and efficiently as possible. [Learn about llm-d](/en/topics/ai/what-is-llm-d) Why is the vLLM open source community so popular? ------------------------------------------------- vLLM has helped address the core issues around efficient GPU utilization, unlocking lower cost per token, stable latency at scale, and doing it with an open, portable deployment approach. That’s why the vLLM community is active and vibrant. Contributions come from passionate groups like Hugging Face, UC Berkeley, NVIDIA, Red Hat, and many more. The community consistently challenges and improves the software in the open source project. With Day 0 support for all major models and accelerators, its accessibility is attractive to both industries and academia. At the start of 2026, the GitHub community supports: * 500K+ GPUs deployed 24/7. * 200+ different accelerator types. * 500+ support model architectures. * 2.2K+ unique contributors. [Join the community on GitHub](https://github.com/sponsors/vllm-project) How Red Hat can help -------------------- [Red Hat® AI](/en/products/ai) is built for fast, flexible, and efficient inference through its [vLLM-powered](/en/topics/ai/what-is-vllm) server. It reliably connects models to your data to unify the customization and development of specialized agents on a single platform. Built on an open source foundation, our products give you full control of AI workflows from end-to-end at any scale. The Red Hat AI portfolio includes [Red Hat AI Inference](/en/products/ai/inference), an inference stack that provides the operational control to run any model on any accelerator across the hybrid cloud. Get fast, efficient, and cost-effective inference at scale. [Explore Red Hat AI](/en/products/ai) Use case Generative AI use cases with Red Hat AI --------------------------------------- In an enterprise setting, generative AI provides an opportunity to boost productivity, improve customer experience, and optimize workflow processes. [Learn more about this use case](/en/products/ai/generative-ai "Generative AI use cases with Red Hat AI") The adaptable enterprise: Why AI readiness is disruption readiness ------------------------------------------------------------------ This e-book, written by Michael Ferris, Red Hat COO and CSO, navigates the pace of change and technological disruption with AI that faces IT leaders today. [Get the resource](/en/resources/the-adaptable-enterprise-ai-ebook "The adaptable enterprise: Why AI readiness is disruption readiness") Keep reading ------------ ### What is agentic AI? Agentic AI is a software system designed to interact with data and tools in a way that requires minimal human intervention. [Read the article](/en/topics/ai/what-is-agentic-ai "article | What is agentic AI?") ### What is generative AI? Generative AI is a kind of artificial intelligence technology that relies on deep learning models trained on large data sets to create new content. [Read the article](/en/topics/ai/what-is-generative-ai "article | what is generative ai?") ### What are large language models? A large language model (LLM) is a type of artificial intelligence that uses machine learning techniques to understand and generate human language. [Read the article](/en/topics/ai/what-are-large-language-models "article | What are large language models?") Artificial intelligence resources --------------------------------- ### Featured product * #### [Red Hat OpenShift AI](/en/products/ai/openshift-ai) An artificial intelligence (AI) platform that provides tools to rapidly develop, train, serve, and monitor models and AI-enabled applications. [See all products](/en/products "See all products") ### Related content * Blog post [The future of AI demands a hybrid foundation](/en/blog/future-ai-demands-hybrid-foundation) * Blog post [AI in production at the industrial edge: A repeatable path with Red Hat and Intel](/en/blog/ai-production-industrial-edge-repeatable-path-red-hat-and-intel) * Blog post [Fragnesia and friends: When page cache vulnerabilities keep coming back](/en/blog/fragnesia-and-friends-when-page-cache-vulnerabilities-keep-coming-back) * E-book [Generative AI in action](/en/resources/generative-ai-in-action-ebook) ### Related articles * [What is agentic AI?](/en/topics/ai/what-is-agentic-ai) * [LoRA vs. QLoRA](/en/topics/ai/lora-vs-qlora) * [SLMs vs LLMs: What are small language models?](/en/topics/ai/llm-vs-slm) * [What are large language models?](/en/topics/ai/what-are-large-language-models) * [What are Granite models?](/en/topics/ai/what-are-granite-models) * [What is Mixture of Experts (MoE)?](/en/topics/ai/mixture-of-experts) * [Predictive AI vs generative AI](/en/topics/ai/predictive-ai-vs-generative-ai) * [What is generative AI?](/en/topics/ai/what-is-generative-ai) * [What are intelligent applications?](/en/topics/ai/what-are-intelligent-applications) * [How vLLM accelerates AI inference: 3 enterprise use cases](/en/topics/ai/how-vllm-accelerates-ai-inference-3-enterprise-use-cases) * [What is AI inference?](/en/topics/ai/what-is-ai-inference) * [vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama) * [RAG vs. fine-tuning](/en/topics/ai/rag-vs-fine-tuning) * [What is AgentOps?](/en/topics/ai/agentops) * [What is parameter-efficient fine-tuning (PEFT)?](/en/topics/ai/what-is-peft) * [AIOps explained](/en/topics/ai/what-is-aiops) * [What is sovereign AI?](/en/topics/ai/sovereign-ai) * [What is MLOps?](/en/topics/ai/what-is-mlops) * [What is Model Context Protocol (MCP)?](/en/topics/ai/what-is-model-context-protocol-mcp) * [What is distributed inference?](/en/topics/ai/what-is-distributed-inference) * [What is retrieval-augmented generation?](/en/topics/ai/what-is-retrieval-augmented-generation) * [Understanding AI/ML use cases](/en/topics/ai/ai-ml-use-cases) * [What is AI in the public sector?](/en/topics/ai/what-is-ai-in-the-public-sector) * [AI infrastructure explained](/en/topics/ai/ai-infrastructure-explained) * [What is AI in healthcare?](/en/topics/ai/what-is-ai-in-healthcare) * [What is explainable AI?](/en/topics/ai/what-explainable-ai) * [Understanding AI in telecommunications with Red Hat](/en/topics/ai/understanding-ai-in-telecommunications) * [What is machine learning?](/en/topics/ai/what-is-machine-learning) * [Agentic AI vs. generative AI](/en/topics/ai/agentic-ai-vs-generative-ai) * [What is LLMops](/en/topics/ai/llmops) * [What are foundation models for AI?](/en/topics/ai/what-are-foundation-models) * [What is deep learning?](/en/topics/ai/what-is-deep-learning) * [What is enterprise AI?](/en/topics/ai/what-is-enterprise-ai) * [What is Model-as-a-Service?](/en/topics/ai/what-is-models-as-a-service) * [What is AI security?](/en/topics/ai/what-is-ai-security) * [What is an AI platform?](/en/topics/ai/what-is-an-ai-platform) * [What is llm-d?](/en/topics/ai/what-is-llm-d) * [What is edge AI?](/en/topics/edge-computing/what-is-edge-ai) * [AI in banking](/en/topics/ai/ai-in-banking) * [What is InstructLab?](/en/topics/ai/what-is-instructlab) * [Why choose Red Hat Ansible Automation Platform as your AI foundation?](/en/topics/automation/automation-and-ai) * [Edge solutions for real-time decision making](/en/topics/edge-computing/edge-solutions-real-time-decision-making) * [What are predictive analytics](/en/topics/automation/how-predictive-analytics-improve-it-performance) * [How Kubernetes can help AI/ML](/en/topics/cloud-computing/how-kubernetes-can-help-ai) * [What is Kubeflow?](/en/topics/cloud-computing/what-is-kubeflow) * [Accelerate MLOps with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/aiml) * [What is a data lake?](/en/topics/data-storage/what-is-a-data-lake) [More about this topic](/en/topics/ai "More about this topic")