* [Topics](/en/topics "Topics") * How vLLM accelerates AI inference: 3 enterprise use cases How vLLM accelerates AI inference: 3 enterprise use cases ========================================================= Published May 12, 2026•*6*-minute read Copy URL Jump to section --------------- What is vLLM?Why does vLLM matter?How does vLLM impact GPUs?Why are companies using vLLM?3 vLLM use casesvLLM’s future impactHow Red Hat can help What is vLLM? ------------- [vLLM](/en/topics/ai/what-is-vllm) is an [inference](/en/topics/ai/what-is-ai-inference) server that speeds up [gen AI](/en/topics/ai/what-is-generative-ai) inference in [large language models (LLMs)](/en/topics/ai/what-are-large-language-models) by making better use of memory storage and graphics processing units (GPUs). Using GPUs more efficiently helps LLMs perform calculations faster and at scale. This becomes increasingly important when organizations need real-time applications like chatbots or multimodal workflows. This article highlights 3 real-world examples of how well-known companies are successfully using vLLM. [Get an in-depth overview of vLLM](/en/topics/ai/what-is-vllm) Why does vLLM matter for AI inference? -------------------------------------- During inference, LLMs rely on key values to do a lot of math in a short period of time. LLMs use key values to attach a numerical value to tokens (terms or phrases) to understand language and calculate answers. So, every token (key) is associated with a number (value) that allows the LLM to calculate a response. AI inference uses key values during its 2 main phases: * **Prefill** is when the model processes the input prompt. The key values for each token create the key value (KV) cache, which serves as the model’s short-term memory. * **Decode** is when the model generates new tokens. It uses the existing KV cache to calculate the key values of a response. LLMs store key values for every processed token in the KV cache. Since the cache grows according to prompt length and output generation, it takes up a lot of LLM memory storage. Traditional LLM memory management systems don’t organize calculations or use memory in the most efficient way, causing LLMs to move slowly. vLLM uses a memory management technique that understands how KV cache is used during inference. It retrieves cache data in a way that identifies repetitive key values to help prevent memory fragmentation and reduce extra work for the LLM. This makes GPU memory usage more efficient and LLM inference faster. [Why you should care about AI inference](/en/artificial-intelligence/inference) 4 key considerations for implementing AI technology --------------------------------------------------- [Get the resource](/en/resources/4-considerations-for-implementing-ai-technology-checklist "Get the resource") How does vLLM decrease GPU storage needs? ----------------------------------------- vLLM uses different technologies and techniques to use less storage and make inference faster: * **Continuous batching** is when LLMs begin the inference process for the next batch of tokens, even if they haven’t finished calculating a prior token. (vLLM can multitask.) * **PagedAttention** is a breakthrough technology that uses the KV cache to remember previous tokens and lean on its memory to save GPU storage. * **Speculative decoding** uses a smaller, faster model to predict incoming tokens, which increases the speed and efficiency of the prefill stage. * **Quantization** is the process of squeezing larger model parameters into smaller formats to reduce storage needs without sacrificing accuracy. There are various [quantization methods](https://developers.redhat.com/articles/2025/08/18/optimizing-generative-ai-models-quantization) for model customization. Processing fewer tokens or generating a response a few seconds faster might seem inconsequential. But when enterprises use this memory-saving technique—across thousands of AI workloads, GPUs, and inference server calculations—it can save significant time, money, and resources. This is a game changer for organizations that want to scale AI at the enterprise level. [See how distributed inference speeds up AI at scale](/en/topics/ai/what-is-distributed-inference) Why are companies using vLLM? ----------------------------- Organizations are using AI inference in high-volume, high-variable workloads. But deploying LLMs consistently at scale requires a lot of computing power, resources, and specialized [operational skills](/en/products/ai/operationalize-ai). vLLM can overcome these challenges by making more efficient use of the hardware needed to support AI inference in the enterprise. This is why vLLM is especially attractive to industries that need flexibility and control in addition to speed. [Read about the benefits of scaling AI](/en/topics/ai/what-is-enterprise-ai) As an open source solution, vLLM allows companies to: * Own and manage their GPUs. * Control their data. * Experiment with new models as soon as they’re released. This level of freedom offers a lower cost per token and fewer privacy concerns. vLLM can be deployed across a variety of hardware including NVIDIA and AMD GPUs, Google TPUs, Intel Gaudi, and AWS Neuron. vLLM also isn’t restricted to specific hardware, meaning it works across the cloud, in the data center, or at the edge. [vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama) vLLM use cases at the enterprise level -------------------------------------- From recruiting efforts to online gaming, scaling inference can become complex quickly. The following examples show how enterprises are using the open source project, vLLM. These companies aren’t Red Hat customers but benefit from the broader vLLM community and the technology it produces. ### How does Roblox use vLLM? Roblox is an online gaming platform that hosts millions of users around the world. Users can create their own gaming experience and play games others have created. Its latest feature, Assistant, an AI chatbot that helps create content, has increased the number of tokens processed to more than 1 billion per week. Additional features such as [real-time AI chat translation](https://corp.roblox.com/newsroom/2024/02/breaking-down-language-barriers-with-a-multilingual-translation-model) and its [voice safety model](https://corp.roblox.com/newsroom/2024/07/deploying-ml-for-voice-safety) have also added inference complexity. This multimodality across millions of user interactions leads to more tokens to process, which requires more resources for inference. To handle the increasing processing demands, Roblox adopted vLLM as its primary inference engine. Roblox specifically leans on vLLM’s speculative decoding capabilities for language tasks to serve its global customer base. Since adopting vLLM, Roblox has experienced a 50% reduction in latency to serve 4 billion tokens per week. vLLM allows Roblox to scale and meet user demand as its platform continues to grow. Roblox chose vLLM because it aligns with its commitment to supporting open source technologies. Listen to Roblox break down how they use vLLM in [Red Hat’s vLLM office hours](https://www.youtube.com/watch?v=-WqJ8PbnFg8). [Read more about Roblox and vLLM](https://corp.roblox.com/newsroom/2024/09/running-ai-inference-at-scale-in-the-hybrid-cloud) ### How does LinkedIn use vLLM? LinkedIn adopted vLLM to support its wide range of gen AI use cases that cater to its large and active audience. As 1 of the world’s largest professional networking sites, LinkedIn hosts more than 1 billion members in more than 200 countries. Now, vLLM allows LinkedIn to support more than 50 gen AI use cases, such as [LinkedIn Hiring Assistant](https://www.linkedin.com/blog/engineering/generative-ai/the-tech-behind-the-first-agent-from-linkedin-hiring-assistant). Using complex classification calculations, LinkedIn Hiring Assistant filters applicant qualifications like years of experience, skills, and previous employment. This helps recruiters match applicants to the best job fit. But processing these wide-ranging classifications requires a lot of tokens (an average of 1,000 per candidate), and applicant pools can fill up with thousands of candidates. More than 50% of applications share prefix tokens (qualifications share similarities). This makes LinkedIn Hiring Assistant a perfect use case for vLLM’s PagedAttention technology and continuous batching capabilities, which both reduce latency, prioritize high throughput, and lower the pressure on GPU storage. Time Per Output Token (TPOT) reflects the average time it takes for a model to generate each individual token. So far, vLLM has helped LinkedIn improve its TPOT by 7%. [Read more about LinkedIn and vLLM](https://www.linkedin.com/blog/engineering/ai/how-we-leveraged-vllm-to-power-our-genai-applications)  ### How does Amazon use vLLM? [Rufus](https://www.aboutamazon.com/news/retail/amazon-rufus), Amazon’s gen AI shopping assistant, aims to improve customer experience by decreasing decision fatigue. Rufus reportedly served [250 million customers in 2025](https://www.genrise.ai/post/amazon-rufus-ai-2025#:~:text=Commercial%20Impact%20in%202025&text=Adoption:%20~250M%20customers%20have%20used,as%20per%20Amazon%20quarterly%20results.&text=Conversion:%20Rufus%20users%20are%20~60%25%20more%20likely%20to%20purchase.&text=Revenue%20trajectory:%20Amazon%20Management%20says,on%20revenue%2C%20not%20just%20visibility.), and that number continues to grow. With a high number of customers using the gen AI shopping assistant, inference complexity increased. Amazon realized no single chip or instance had enough memory for Rufus to run smoothly. Amazon prioritized scalable, multinode inference capabilities that maintain accuracy at faster speeds and lower latency. They achieved this by combining a multinode architecture solution that integrated with vLLM for smoother, faster inference. By using vLLM’s continuous batching technique, the multinode architecture was able to intelligently schedule inference processing so token volume didn’t impact latency or performance. Using vLLM to increase the efficiency and throughput of its LLMs allows Amazon to scale gen AI projects like Rufus that will continue to grow and evolve with its customers. [Read more about Amazon and vLLM](https://aws.amazon.com/blogs/machine-learning/how-amazon-scaled-rufus-by-building-multi-node-inference-using-aws-trainium-chips-and-vllm/#:~:text=To%20create%20a%20multi%2Dnode,with%20the%20AWS%20Neuron%20SDK.) How will vLLM impact the future of inference? --------------------------------------------- vLLM continues to be the foundation for the future of AI inference due to its core capabilities: * **Speed:** Inference capabilities are constantly improving. vLLM’s hardware and model providers contribute directly to the project to improve speed and model efficiency. * **Community:** vLLM has a large open source community that continues to grow. All of the top 10 model contributors—such as Deepseek, NVIDIA, Meta, and Google—are creating models prebuilt for vLLM because of its efficiency. * **Flexibility:** vLLM can be deployed across most AI hardware, including NVIDIA and AMD GPUs, Google TPUs, Intel Gaudi, AWS Neuron, and other accelerators like MetaX, Rebellions, and more. The diverse hardware support gives enterprises the flexibility they need to deliver outcomes with resources they already have. * **Day-zero support:** When popular model builders like Meta or Google release a new model, vLLM is already familiar with its existing architectures. This means vLLM can offer day-zero (immediate) support for new models. So, vLLM is an accessible, out-of-the-box solution for enterprises that want to speed up their model deployment and lower costs. vLLM also includes [llm-d](/en/topics/ai/what-is-llm-d), a distributed inference framework for managing LLMs at scale in the hybrid cloud. [Explore the vLLM community on GitHub](https://github.com/vllm-project/vllm) How Red Hat can help -------------------- [Red Hat® AI](/en/products/ai) is built for fast, flexible, and efficient inference through its [vLLM-powered](/en/topics/ai/what-is-vllm) server. It reliably connects models to your data to unify the customization and development of specialized agents on a single platform. Built on an open source foundation, our products give you full control of AI workflows from end-to-end at any scale. The Red Hat AI portfolio includes [Red Hat AI Inference](/en/products/ai/inference), an inference stack that provides the operational control to run any model on any accelerator across the hybrid cloud. Get fast, efficient, and cost-effective inference at scale. [Explore Red Hat AI](/en/products/ai) Campaign solution Artificial intelligence (AI) at Red Hat --------------------------------------- From live events to hands-on product demos to deep technical research, see what we're doing with AI at Red Hat. [Keep reading](/en/artificial-intelligence "Artificial intelligence (AI) at Red Hat") Get started with AI for enterprise organizations: A beginner’s guide -------------------------------------------------------------------- Explore this beginner's guide to find out how Red Hat OpenShift AI and Red Hat Enterprise Linux AI can accelerate your AI adoption journey. [Get the resource](/en/resources/artificial-intelligence-for-enterprise-beginners-guide-ebook "Get started with AI for enterprise organizations: A beginner’s guide") Keep reading ------------ ### What is agentic AI? Agentic AI is a software system designed to interact with data and tools in a way that requires minimal human intervention. [Read the article](/en/topics/ai/what-is-agentic-ai "article | What is agentic AI?") ### What is generative AI? Generative AI is a kind of artificial intelligence technology that relies on deep learning models trained on large data sets to create new content. [Read the article](/en/topics/ai/what-is-generative-ai "article | what is generative ai?") ### What are large language models? A large language model (LLM) is a type of artificial intelligence that uses machine learning techniques to understand and generate human language. [Read the article](/en/topics/ai/what-are-large-language-models "article | What are large language models?") Artificial intelligence resources --------------------------------- ### Featured product * #### [Red Hat AI](/en/products/ai) Flexible solutions that accelerate AI solution development and deployment across hybrid cloud environments. [See all products](/en/technologies/all-products "See all products") ### Related content * Blog post [A decade of open innovation: Red Hat continues to scale the open hybrid cloud with Microsoft](/en/blog/decade-open-innovation-red-hat-continues-scale-open-hybrid-cloud-microsoft) * Blog post [Discover the Red Hat OpenShift AI model catalog](/en/blog/discover-red-hat-openshift-ai-model-catalog) * Blog post [Get a custom-made learning plan with Red Hat Guided Learning](/en/blog/get-custom-made-learning-plan-red-hat-guided-learning) * Blog post [Agentic AI demands a new infrastructure stack: AMD and Red Hat deliver](/en/blog/agentic-ai-demands-new-infrastructure-stack-amd-and-red-hat-deliver) ### Related articles * [What is agentic AI?](/en/topics/ai/what-is-agentic-ai) * [LoRA vs. QLoRA](/en/topics/ai/lora-vs-qlora) * [What are large language models?](/en/topics/ai/what-are-large-language-models) * [What are Granite models?](/en/topics/ai/what-are-granite-models) * [What is Mixture of Experts (MoE)?](/en/topics/ai/mixture-of-experts) * [Predictive AI vs generative AI](/en/topics/ai/predictive-ai-vs-generative-ai) * [What is generative AI?](/en/topics/ai/what-is-generative-ai) * [What are intelligent applications?](/en/topics/ai/what-are-intelligent-applications) * [What is AI inference?](/en/topics/ai/what-is-ai-inference) * [vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama) * [RAG vs. fine-tuning](/en/topics/ai/rag-vs-fine-tuning) * [What is vLLM?](/en/topics/ai/what-is-vllm) * [SLMs vs LLMs: What are small language models?](/en/topics/ai/llm-vs-slm) * [What is AgentOps?](/en/topics/ai/agentops) * [What is parameter-efficient fine-tuning (PEFT)?](/en/topics/ai/what-is-peft) * [AIOps explained](/en/topics/ai/what-is-aiops) * [What is Model Context Protocol (MCP)?](/en/topics/ai/what-is-model-context-protocol-mcp) * [What is retrieval-augmented generation?](/en/topics/ai/what-is-retrieval-augmented-generation) * [Understanding AI/ML use cases](/en/topics/ai/ai-ml-use-cases) * [What is distributed inference?](/en/topics/ai/what-is-distributed-inference) * [AI infrastructure explained](/en/topics/ai/ai-infrastructure-explained) * [What is AI in healthcare?](/en/topics/ai/what-is-ai-in-healthcare) * [What is AI in the public sector?](/en/topics/ai/what-is-ai-in-the-public-sector) * [What is explainable AI?](/en/topics/ai/what-explainable-ai) * [Understanding AI in telecommunications with Red Hat](/en/topics/ai/understanding-ai-in-telecommunications) * [What is machine learning?](/en/topics/ai/what-is-machine-learning) * [What is LLMops](/en/topics/ai/llmops) * [What are foundation models for AI?](/en/topics/ai/what-are-foundation-models) * [What is deep learning?](/en/topics/ai/what-is-deep-learning) * [Agentic AI vs. generative AI](/en/topics/ai/agentic-ai-vs-generative-ai) * [What is MLOps?](/en/topics/ai/what-is-mlops) * [What is sovereign AI?](/en/topics/ai/sovereign-ai) * [What is enterprise AI?](/en/topics/ai/what-is-enterprise-ai) * [What is Model-as-a-Service?](/en/topics/ai/what-is-models-as-a-service) * [What is an AI platform?](/en/topics/ai/what-is-an-ai-platform) * [What is AI security?](/en/topics/ai/what-is-ai-security) * [What is llm-d?](/en/topics/ai/what-is-llm-d) * [What is edge AI?](/en/topics/edge-computing/what-is-edge-ai) * [AI in banking](/en/topics/ai/ai-in-banking) * [What is InstructLab?](/en/topics/ai/what-is-instructlab) * [Why choose Red Hat Ansible Automation Platform as your AI foundation?](/en/topics/automation/automation-and-ai) * [Edge solutions for real-time decision making](/en/topics/edge-computing/edge-solutions-real-time-decision-making) * [What are predictive analytics](/en/topics/automation/how-predictive-analytics-improve-it-performance) * [How Kubernetes can help AI/ML](/en/topics/cloud-computing/how-kubernetes-can-help-ai) * [What is Kubeflow?](/en/topics/cloud-computing/what-is-kubeflow) * [Accelerate MLOps with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/aiml) * [What is a data lake?](/en/topics/data-storage/what-is-a-data-lake) [More about this topic](/en/topics/ai "More about this topic")