* [Topics](/en/topics "Topics")
* [Artificial intelligence](/en/topics/ai "Artificial intelligence")
* What is AI inference?
What is AI inference?
=====================
Updated  May 12, 2026•*8*-minute read
Copy URL
Jump to section
---------------
What is AI inference?Why is AI inference important?AI inference use casesWhat is AI training?Types of AI inferenceWhat is an AI inference server? AI inference challengesHow vLLM helps LLMs inference fasterHow MoE helps LLMs inference fasterDistributed inferenceHow Red Hat can help
What is AI inference?
---------------------
AI inference is when an AI model provides an answer based on data. What some generally call “AI” is really the success of AI inference: the final step—the “aha” moment—in a long and complex process of [machine learning](/en/topics/ai/what-is-machine-learning) technology.
Training artificial intelligence (AI) models with sufficient data can help improve AI inference accuracy and speed.
[Why you should care about AI inference](/en/artificial-intelligence/inference)
For example, when an AI model is trained on data about animals—from their differences and similarities to typical health and behavior—it needs a large data set to make connections and identify patterns.
After successful training, the model can make inferences such as identifying a breed of dog, recognizing a cat’s meow, or even delivering a warning around a spooked horse. Even though it has never seen these animals outside of an abstract data set before, the extensive data it was trained on allows it to make inferences in a new environment in real time.
Our own human brain makes connections like this too. We can read about different animals from books, movies, and online resources. We can see pictures, watch videos, and listen to what these animals sound like. When we go to the zoo, we are able to make an inference (“That’s a buffalo!”). Even if we have never been to the zoo, we can identify the animal because of the research we have done. The same goes for AI models during AI inference.
Find out what's new and what's next for Red Hat AI at our next live event. [Catch the next live session.](/en/events/ai)
[Explore Red Hat AI](/en/products/ai)
Why is AI inference important?
------------------------------
AI inference is the operational phase of AI, where the model is able to apply what it’s learned from training to real-world situations. AI’s ability to identify patterns and reach conclusions sets it apart from other technologies. Its ability to infer can help with practical day-to-day tasks or extremely complicated computer programming.
But inference gets a lot of pressure from models that keep growing bigger. As models get more complex, inference becomes slower.
For inference to be successful, AI models need to do a lot of math in a short period of time. So, factors like model size, high user volume, and latency can all limit performance. When models require more data and more memory, hardware and accelerators struggle to keep up.
That's why the hardware and software that support your inference capabilities can make or break your AI strategy.
[Learn more about why inference is so important](/en/artificial-intelligence/inference)
Red Hat AI
----------
[Learn more about this product](/en/products/ai "Learn more about this product")
AI inference use cases
----------------------
Today, businesses can use AI inference in a variety of everyday use cases. These are a few examples:
**Healthcare**: AI inference can [help healthcare professionals](/en/topics/ai/what-is-ai-in-healthcare) compare patient history to current data and trace patterns and anomalies faster than humans. This could be an outlier on a brain scan or an extra “thump” in a heart beat. This can help catch signs of threats to patient health much earlier, and much faster.
**Finance**: After being trained on large data sets of [banking](/en/topics/ai/ai-in-banking) and credit information, AI inference can identify errors or unusual data in real-time to catch fraud early and quickly. This can optimize customer service resources, protect customer privacy, and improve brand reputation.
**Automotive:** As AI enters the world of cars, [autonomous vehicles](/en/topics/edge-computing/what-is-an-autonomous-vehicle) are changing the way we drive. AI inference can help vehicles navigate the most efficient route from point A to point B or brake when they approach a stop sign, all to improve the ease and the safety of those in the car.
[Predictive AI vs. generative AI](/en/topics/ai/predictive-ai-vs-generative-ai)
Many other industries are applying AI inference in creative ways, too. It can be applied to a fast food drive-through, a veterinary clinic, or a hotel concierge. Businesses are finding ways to make this technology work to their advantage to improve their accuracy, save time and money, and maintain their edge with competitors.
 [More AI/ML use cases](/en/topics/ai/ai-ml-use-cases)
What is AI training?
--------------------
AI training is the process of using data to teach the model how to make connections and identify patterns. Training is the process of teaching a model, whereas inference is the AI model in action.
[What are foundation models?](/en/topics/ai/what-are-foundation-models)
Most AI training occurs in the beginning stages of model building. Once trained, the [model can make connections](/en/topics/ai/what-is-machine-learning) with data it has never encountered before. Training an AI model with a larger data set means it can learn more connections and make more accurate inferences. If the model is struggling to make accurate inferences after training, [fine-tuning](/en/topics/ai/rag-vs-fine-tuning) can add knowledge and improve accuracy.
Training and AI inference are how AI is able to mimic human capabilities such as drawing conclusions based on evidence and reasoning.
Factors like model size can change the amount of resources you need to manipulate your model.
[Learn how smaller models can make GPU inference easier.](https://neuralmagic.com/blog/24-sparse-llama-smaller-models-for-efficient-gpu-inference/)
What are different types of AI inference?
-----------------------------------------
Different kinds of AI inference can support different use cases.
* **Batch inference:** Batch inference gets its name from how it receives and processes data: in large groups. Instead of processing inference in real time, this method processes data in waves, sometimes hourly or even daily, depending on the amount of data and the efficiency of the AI model. These inferences can also be called “offline inferences” or “static inferences.”
* **Online inference:** Online inference or “dynamic” inference can deliver a response in real time. These inferences require hardware and software that can reduce latency barriers and support high-speed predictions. Online inference is helpful [at the edge](/en/topics/edge-computing/what-is-edge-ai), meaning AI is doing its work where the data is located. This could be on a phone, in a car, or at a remote office with limited connectivity.
  OpenAI’s ChatGPT is a good example of an online inference—it requires a lot of upfront operational support in order to deliver a quick and accurate response.
* **Streaming inference:** Streaming inference describes an AI system that is not necessarily used to communicate with humans. Instead of prompts and requests, the model receives a constant flow of data in order to make predictions and update its internal database. Streaming inference can monitor changes, maintain regularity, or predict an issue before it arises.
[Learn how distributed inference with vLLM can alleviate bottlenecks](https://neuralmagic.com/blog/distributed-inference-with-vllm/)
What is an AI inference server?
-------------------------------
An AI inference server is the software that helps an AI model make the jump from training to operating. It uses machine learning to help the model apply what it’s learned and put it into practice to generate inferences.
For efficient results, your AI inference server and AI model need to be compatible. Here are a few examples of inference servers and the models they work with best:
* **Multimodal inference server:** This type of inference server is able to support several models at once. This means it can receive data in the form of code, images, or text and process all of these different inferences on a single server. A multimodal inference server uses GPU and CPU memory more efficiently to support more than one model. This helps streamline hardware, makes it easier to scale, and optimizes costs.
* **Single-model inference server:**This inference server only supports one model, rather than several. The AI inference process is specialized to communicate with a model trained on a specific use case. It may only be able to process data in the form of text or only in the form of code. Its specialized nature allows it to be incredibly efficient, which can help with real-time decision making or resource constraints.
[Find benefits and use cases of fast, efficient inference](/en/products/ai/fast-efficient-inference)
AI inference challenges
-----------------------
The biggest challenges when running AI inference are scaling, resources, and cost.
* **Complexity:** It is easier to teach a model to execute simple tasks like generating a picture or informing a customer of a return policy. As we lean on models to learn more complex data—like how to catch financial fraud or identify medical anomalies—they require more data during training and more resources to support that data.
* **Resources:** More complex models will require specialized hardware and software to support the vast amount of data processing which takes place when a model is generating inferences. A key component of these resources is central processing unit (CPU) memory. A CPU is often referred to as the hub or control center of a computer. When a model is preparing to use what it knows (training data) to generate an answer, it must refer back to the data which is held in CPU memory space.
* **Cost:** All of these puzzle pieces that make AI inference possible are not cheap. Whether your goal is to scale or to transition to the latest AI-supported hardware, the resources it takes to get the full picture can be extensive. As model complexity increases and hardware continues to evolve, costs can increase sharply and make it tough for organizations to keep up with AI innovation.
This video can't play due to privacy settings
To change your settings, select the "Cookie Preferences" link in the footer and opt in to "Advertising Cookies or try disabling adblockers."
A specific inference engine known as vLLM, helps keep these challenges at bay. vLLM speeds up the output of generative AI applications by making better use of the GPU memory. vLLM is a library of open source code maintained by the vLLM community. It helps large language models (LLMs) perform calculations more efficiently and at scale. It uses tools like [LLM Compressor](https://neuralmagic.com/blog/llm-compressor-is-here-faster-inference-with-vllm/) to help you inference faster, taking a big burden off of your team and resources.
[What is vLLM?](/en/topics/ai/what-is-vllm)
How vLLM helps LLMs inference faster
------------------------------------
AI inference is being used in high-volume, high-variable use cases. But deploying LLMs consistently at scale requires a lot of computing power, resources, and specialized operational skills. vLLM can solve these challenges by making more efficient use of the hardware needed to support AI inference in the enterprise. This is why vLLM is especially attractive to industries that need flexibility and control in addition to speed.
[vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama)
As an open source solution, vLLM allows companies to:
* Own and manage their GPUs.
* Control their data.
* Experiment with state-of-the-art models as soon as they are released.
[vLLM](/en/topics/ai/what-is-vllm) can be deployed across a variety of hardware including NVIDIA and AMD GPUs, Google TPUs, Intel Gaudi, and AWS Neuron. vLLM is also not restricted to specific hardware, meaning it works across the cloud, in the data center, or at the edge.
Learn how well-known organizations are using vLLM to scale effectively in these 3 real-world use cases.
[Explore 3 real-world vLLM use cases](/en/topics/ai/how-vllm-accelerates-ai-inference-3-enterprise-use-cases)
How MoE helps LLMs inference faster
-----------------------------------
To overcome challenges like latency and resource constraints, Mixture of Experts (MoEs) creates a neural network that supports faster inference at scale.
MoE is a model architecture technique that speeds up AI inference by routing tasks to the most capable part of the model.
MoE models are specially trained to answer specific subcategories quickly and accurately. It uses techniques like sparsity in its underlying neural network and deep learning architecture. This helps LLMs move faster and more accurately at scale.
[Learn more about MoE and how to speed up inference](/en/topics/ai/mixture-of-experts)
What is distributed inference?
------------------------------
[Distributed inference](/en/topics/ai/what-is-distributed-inference) lets AI models process workloads more efficiently by dividing the labor of inference across a group of interconnected devices. Think of it as the software equivalent of the saying, “many hands make light work.”
Distributed inference supports a system that splits requests across a fleet of hardware, which can include physical and cloud servers. From there, each inference server processes its assigned portion in parallel to create an output. The result is a resilient and observable system for delivering consistent and scalable AI-powered services.
Distributed inference is supported on vLLM, using techniques like tensor parallelism and Mixture-of-Experts (MOE) architectures.
[Find out more about distributed inference](/en/topics/ai/what-is-distributed-inference)
How Red Hat can help
--------------------
[Red Hat® AI](/en/products/ai) is built for fast, flexible, and efficient inference through its [vLLM-powered](/en/topics/ai/what-is-vllm) server. It reliably connects models to your data to unify the customization and development of specialized agents on a single platform. Built on an open source foundation, our products give you full control of AI workflows from end-to-end at any scale.
The Red Hat AI portfolio includes [Red Hat AI Inference](/en/products/ai/inference), an inference stack that provides the operational control to run any model on any accelerator across the hybrid cloud. Get fast, efficient, and cost-effective inference at scale.
[Explore Red Hat AI](/en/products/ai)
Press release
Red Hat Announces Definitive Agreement to Acquire Neural Magic
--------------------------------------------------------------
Red Hat announced that it has signed a definitive agreement to acquire Neural Magic, a pioneer in software and algorithms that accelerate generative AI (gen AI) inference workloads.
[Read the press release](/en/about/press-releases/red-hat-acquire-neural-magic "Red Hat Announces Definitive Agreement to Acquire Neural Magic")
The adaptable enterprise: Why AI readiness is disruption readiness
------------------------------------------------------------------
This e-book, written by Michael Ferris, Red Hat COO and CSO, navigates the pace of change and technological disruption with AI that faces IT leaders today.
[Get the resource](/en/resources/the-adaptable-enterprise-ai-ebook "The adaptable enterprise: Why AI readiness is disruption readiness")
Keep reading
------------
### What is agentic AI?
Agentic AI is a software system designed to interact with data and tools in a way that requires minimal human intervention.
[Read the article](/en/topics/ai/what-is-agentic-ai "article | What is agentic AI?")
### What is generative AI?
Generative AI is a kind of artificial intelligence technology that relies on deep learning models trained on large data sets to create new content.
[Read the article](/en/topics/ai/what-is-generative-ai "article | what is generative ai?")
### What are large language models?
A large language model (LLM) is a type of artificial intelligence that uses machine learning techniques to understand and generate human language.
[Read the article](/en/topics/ai/what-are-large-language-models "article | What are large language models?")
Artificial intelligence resources
---------------------------------
### Related content
* Blog post
  [The future of AI demands a hybrid foundation](/en/blog/future-ai-demands-hybrid-foundation)
* Blog post
  [Accelerate autoscaling inference in Red Hat AI with Everpure](/en/blog/accelerate-autoscaling-inference-red-hat-ai-everpure)
* Blog post
  [AI in production at the industrial edge: A repeatable path with Red Hat and Intel](/en/blog/ai-production-industrial-edge-repeatable-path-red-hat-and-intel)
* Blog post
  [Fragnesia and friends: When page cache vulnerabilities keep coming back](/en/blog/fragnesia-and-friends-when-page-cache-vulnerabilities-keep-coming-back)
### Related articles
* [What is agentic AI?](/en/topics/ai/what-is-agentic-ai)
* [What is vLLM?](/en/topics/ai/what-is-vllm)
* [vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama)
* [RAG vs. fine-tuning](/en/topics/ai/rag-vs-fine-tuning)
* [LoRA vs. QLoRA](/en/topics/ai/lora-vs-qlora)
* [SLMs vs LLMs: What are small language models?](/en/topics/ai/llm-vs-slm)
* [What are large language models?](/en/topics/ai/what-are-large-language-models)
* [What are Granite models?](/en/topics/ai/what-are-granite-models)
* [What is Mixture of Experts (MoE)?](/en/topics/ai/mixture-of-experts)
* [Predictive AI vs generative AI](/en/topics/ai/predictive-ai-vs-generative-ai)
* [What is generative AI?](/en/topics/ai/what-is-generative-ai)
* [What are intelligent applications?](/en/topics/ai/what-are-intelligent-applications)
* [How vLLM accelerates AI inference: 3 enterprise use cases](/en/topics/ai/how-vllm-accelerates-ai-inference-3-enterprise-use-cases)
* [What is AgentOps?](/en/topics/ai/agentops)
* [What is parameter-efficient fine-tuning (PEFT)?](/en/topics/ai/what-is-peft)
* [AIOps explained](/en/topics/ai/what-is-aiops)
* [Understanding AI in telecommunications with Red Hat](/en/topics/ai/understanding-ai-in-telecommunications)
* [What is machine learning?](/en/topics/ai/what-is-machine-learning)
* [Agentic AI vs. generative AI](/en/topics/ai/agentic-ai-vs-generative-ai)
* [What is LLMops](/en/topics/ai/llmops)
* [What are foundation models for AI?](/en/topics/ai/what-are-foundation-models)
* [What is deep learning?](/en/topics/ai/what-is-deep-learning)
* [What is MLOps?](/en/topics/ai/what-is-mlops)
* [What is sovereign AI?](/en/topics/ai/sovereign-ai)
* [What is Model Context Protocol (MCP)?](/en/topics/ai/what-is-model-context-protocol-mcp)
* [What is distributed inference?](/en/topics/ai/what-is-distributed-inference)
* [What is retrieval-augmented generation?](/en/topics/ai/what-is-retrieval-augmented-generation)
* [Understanding AI/ML use cases](/en/topics/ai/ai-ml-use-cases)
* [What is AI in the public sector?](/en/topics/ai/what-is-ai-in-the-public-sector)
* [AI infrastructure explained](/en/topics/ai/ai-infrastructure-explained)
* [What is AI in healthcare?](/en/topics/ai/what-is-ai-in-healthcare)
* [What is explainable AI?](/en/topics/ai/what-explainable-ai)
* [What is AI security?](/en/topics/ai/what-is-ai-security)
* [What is an AI platform?](/en/topics/ai/what-is-an-ai-platform)
* [What is enterprise AI?](/en/topics/ai/what-is-enterprise-ai)
* [What is Model-as-a-Service?](/en/topics/ai/what-is-models-as-a-service)
* [What is llm-d?](/en/topics/ai/what-is-llm-d)
* [What is edge AI?](/en/topics/edge-computing/what-is-edge-ai)
* [AI in banking](/en/topics/ai/ai-in-banking)
* [What is InstructLab?](/en/topics/ai/what-is-instructlab)
* [Why choose Red Hat Ansible Automation Platform as your AI foundation?](/en/topics/automation/automation-and-ai)
* [Edge solutions for real-time decision making](/en/topics/edge-computing/edge-solutions-real-time-decision-making)
* [What are predictive analytics](/en/topics/automation/how-predictive-analytics-improve-it-performance)
* [How Kubernetes can help AI/ML](/en/topics/cloud-computing/how-kubernetes-can-help-ai)
* [What is Kubeflow?](/en/topics/cloud-computing/what-is-kubeflow)
* [Accelerate MLOps with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/aiml)
* [What is a data lake?](/en/topics/data-storage/what-is-a-data-lake)
[More about this topic](/en/topics/ai "More about this topic")