* [Topics](/en/topics "Topics") * AI infrastructure explained AI infrastructure explained =========================== Updated April 15, 2026•*5*-minute read Copy URL Jump to section --------------- What is AI infrastructureInfrastructure componentsWhy does your AI infrastructure matterAI infrastructure and inferenceHow Red Hat can help What is AI infrastructure? -------------------------- With [artificial intelligence](/en/topics/ai) (AI) growing in use with our daily lives, it’s crucial to have a structure that allows effective and efficient workflows. That’s where artificial intelligence infrastructure (AI infrastructure) comes in. A well-designed infrastructure helps data scientists and developers access data, deploy machine learning algorithms, and manage the hardware’s computing resources. AI infrastructure combines artificial intelligence and machine learning (AI/ML) technology to develop and deploy reliable and scalable data solutions. It is the technology that enables [machine learning](/en/topics/ai/what-is-machine-learning), allowing machines to think like humans. Machine learning is the technique of training a computer to find patterns, [make predictions](/en/topics/ai/predictive-ai-vs-generative-ai), and learn from experience without being explicitly programmed. It can be applied to [generative AI](/en/topics/ai/what-is-generative-ai), and is made possible through [deep learning](/en/topics/ai/what-is-deep-learning), a machine learning technique for analyzing and interpreting large amounts of data. [Explore Red Hat AI](/en/products/ai) ### AI infrastructure tech stack A tech stack, short for technology stack, is a set of technologies, frameworks, and tools used to build and deploy software applications. As a visual, these technologies “stack” on top of each other to build an application. An AI infrastructure tech stack can enable faster development and deployment of applications through three essential layers. [What does an AI tech stack look like at the enterprise?](/en/topics/ai/what-is-enterprise-ai) The **applications layer** gives humans the opportunity to collaborate with machines when working with tools like end-to-end apps or end-user-facing apps. End-user-facing applications are usually built using open-source AI frameworks to create models that are customizable and can be tailored to meet specific business needs. The **model layer** helps AI products function. This layer requires a hosting solution for deployment. There are three models to this layer that provide a foundation. * *General AI*: Mimics the human brain's ability to think and make decisions. Think of AI apps like ChatGPT and DALL-E from OpenAI. * *Specific AI*: Uses specific data to generate the exact results. Think of tasks like generating ad copy and song lyrics. * *Hyperlocal AI*: the artificial intelligence that can achieve the highest levels of accuracy and relevance, designed to be specialists in their field. Think of writing scientific articles or creating interior design mockups The **infrastructure layer** includes the hardware and software needed to build and train models. Components like specialized processors like GPUs (hardware) and optimization and deployment tools (software) fall under this layer. [Cloud computing](/en/topics/cloud-computing) services are also a part of the infrastructure layer. This video can't play due to privacy settings To change your settings, select the "Cookie Preferences" link in the footer and opt in to "Advertising Cookies or try disabling adblockers." AI infrastructure components ---------------------------- Now that we have covered the three layers involved in an AI infrastructure, let’s explore a few components that are required to build, deploy, and maintain AI models. #### Data storage [Data storage](/en/topics/data-storage) is the collection and retention of digital information—the bits and bytes behind applications, network protocols, documents, media, address books, user preferences, and more. Data storage is important for storing, organizing, and retrieving AI information. #### Data management [Data management](/en/topics/data-services/what-is-data-management) is the process of gathering, storing, and using data, often facilitated by data management software. It allows you to know what data you have, where it is located, who owns it, who can see it, and how it is accessed. With the appropriate controls and implementation, data management workflows deliver the analytical insights needed to make better decisions. #### Optimization software Make your hardware run as efficiently as possible with optimization software like [vLLM](/en/topics/ai/what-is-vllm) and [llm-d](/en/topics/ai/what-is-llm-d). * vLLM, which stands for virtual large language model, is a library of open source code. It helps large language models (LLMs) perform calculations more efficiently and at scale. More specifically, vLLM is an inference server that speeds up the output of generative AI applications by making better use of the GPU memory. * llm-d is a Kubernetes-native, open source framework that speeds up [distributed inference](/en/topics/ai/what-is-distributed-inference) at scale. llm-d builds on the power of vLLM, acting as a coordinator to make processing happen and quickly and efficiently as possible. [What is AI inference?](/en/topics/ai/what-is-ai-inference) #### Machine learning frameworks Machine learning (ML) is a subcategory of artificial intelligence (AI) that uses algorithms to identify patterns and make predictions within a set of data, and the frameworks provide the tools and libraries needed. #### Machine learning operations [Machine learning operations (MLOps)](/en/topics/ai/what-is-mlops) is a set of workflow practices that aims to streamline the process of producing, maintaining, and monitoring machine learning (ML) models. Inspired by [DevOps](/en/topics/devops) and GitOps principles, MLOps seeks to establish a continuous and ever-evolving process for integrating ML models into software development processes. [Learn more about building a AI/ML environment](/en/resources/building-production-ready-ai-environment-ebook "Top considerations for building a production-ready AI/ML environment") Red Hat resources ----------------- [Keep reading](/en/resources "Keep reading") Why does your AI infrastructure matter? --------------------------------------- A solid AI infrastructure with established components contributes to innovation and efficiency. However, there are benefits, challenges, and applications to consider when designing an AI infrastructure. #### Benefits AI infrastructure has several benefits for your AI operations and organizations. One benefit is **scalability**, providing the opportunity to upscale and downscale operations on demand, especially with cloud-based AI/ML solutions. Another benefit is **automation**, allowing repetitive work to decrease errors and increase deliverable turn around times. [What is Model-as-a-Service?](/en/topics/ai/what-is-models-as-a-service) #### Challenges Despite its benefits, AI infrastructure does have some challenges. One of the biggest challenges is the amount and quality of data that needs to be processed. Because AI systems rely on large amounts of data to learn and make decisions, traditional data storage and processing methods may not be enough to handle the scale and complexity of AI workloads. Another big challenge is the requirement for real-time analysis and decision-making. This requirement means that the infrastructure has to process data quickly and efficiently, which needs to be taken into account to integrate the right solution to deal with large volumes of data. [Learn how automation can help](/en/topics/automation/automation-and-ai) #### Applications There are applications that can address these challenges. With Red Hat® OpenShift® [cloud services](/en/technologies/cloud-computing/openshift/openshift-cloud-services), you can build, deploy, and scale applications quickly. You can also enhance efficiency by improving consistency and security with proactive management and support. Red Hat [Edge](/en/products/edge) helps you deploy closer to where data is collected and gain actionable insights. [Learn more about cloud services for AI/ML](/en/resources/openshift-ai-overview "Red Hat OpenShift Data Science: Cloud services for AI/ML") #### Security AI security defends AI applications against malicious attacks that aim to weaken workloads, manipulate data, or steal sensitive information. It adapts principles of confidentiality, integrity, and availability for the AI lifecycle and technical ecosystems. To protect your AI systems, it’s important to understand them inside and out. The more you understand your AI technology and its infrastructure, the better you can protect it. [Learn more about AI security](/en/topics/ai/what-is-ai-security) How does AI infrastructure impact inference? -------------------------------------------- When you're thinking about your AI infrastructure, it's important not to forget about [inference](/en/topics/ai/what-is-ai-inference). Your infrastucture can have a big impact on your inference capabilities. AI infrastructure can impact: * Latency. * Tokens generated per second. * User concurrency. * Costs. An AI infrastructure that doesn't support inference can lead to slower response times, latency bottlenecks, and make it more expensive to scale. That's why the hardware and software that support your inference capabilities can make or break your AI strategy. [Why you should care about AI inference](/en/artificial-intelligence/inference) ### **How does Mixture of Experts architecture speed up inference?** Mixture of Experts (MoE) is a model architecture technique that speeds up AI inference by routing tasks to the most capable part of the model. To overcome challenges like latency and resource constraints, MoE creates a neural network that supports faster inference at scale. MoE models are specially trained to answer specific subcategories quickly and accurately. This helps LLMs move faster and more accurately at scale. A majority of foundation models use a type of neural network known as transformers. They help models capture contextual relationships and dependencies in data sequences. Developers often replace dense architectures with MoEs to make the model more efficient. MoE is made up of 2 main parts: Sparse neural network layers and a gate network. MoE architecture allows multiple specialized models to work together. So oftentimes, the router identifies more than 1 expert that can answer the prompt quickly. After the experts have completed their tasks, the gating network collects the results and combines them for a final, cohesive answer. [Learn more about MoE and how to speed up inference](/en/topics/ai/mixture-of-experts) How Red Hat can help -------------------- [Red Hat® AI](/en/products/ai) is built for fast, flexible, and efficient inference through its [vLLM-powered](/en/topics/ai/what-is-vllm) server. It reliably connects models to your data to unify the customization and development of specialized agents on a single platform. Built on an open source foundation, our products give you full control of AI workflows from end-to-end at any scale. The Red Hat AI portfolio includes [Red Hat AI Enterprise](/en/products/ai/enterprise), a platform for deploying, managing, and scaling AI inference, agentic AI workflows, and AI-powered applications on any infrastructure. [Explore Red Hat AI](/en/products/ai) The official Red Hat blog ------------------------- Get the latest information about our ecosystem of customers, partners, and communities. [Keep reading](/en/blog "The official Red Hat blog") The adaptable enterprise: Why AI readiness is disruption readiness ------------------------------------------------------------------ This e-book, written by Michael Ferris, Red Hat COO and CSO, navigates the pace of change and technological disruption with AI that faces IT leaders today. [Get the resource](/en/resources/the-adaptable-enterprise-ai-ebook "The adaptable enterprise: Why AI readiness is disruption readiness") Keep reading ------------ ### What is Mixture of Experts (MoE)? Mixture of Experts (MoE) is a model architecture technique that speeds up AI inference by routing tasks to the most capable part of the model. [Read the article](/en/topics/ai/mixture-of-experts "What is Mixture of Experts (MoE)?") ### What is AgentOps? AgentOps (agent operations) is a framework of tools for monitoring the "brain" of an AI as it makes decisions in real time. [Read the article](/en/topics/ai/agentops "What is AgentOps?") ### AIOps explained AIOps (AI for IT operations) is an approach to automating IT operations with machine learning and other advanced AI techniques. [Read the article](/en/topics/ai/what-is-aiops "article | AIOps explained") Artificial intelligence resources --------------------------------- ### Related content * Analyst material [Forrester Research Trends Report, Understand Your Agentic AI Platform Options](/en/resources/forrester-agentic-ai-platform-analyst-material) * Analyst material [Forrester Research Trends Report, Understand Your Agentic AI Platform Options](/en/engage/forrester-agentic-ai-platform-analyst-material) * Blog post [Unlocking sovereign AI and protected collaboration with confidential computing](/en/blog/unlocking-sovereign-ai-and-protected-collaboration-confidential-computing) * Blog post [Designing multitenant GPU infrastructure: Isolation across virtualization and Kubernetes platforms](/en/blog/designing-multitenant-gpu-infrastructure-isolation-across-virtualization-and-kubernetes-platforms) ### Related articles * [What is Mixture of Experts (MoE)?](/en/topics/ai/mixture-of-experts) * [What is AgentOps?](/en/topics/ai/agentops) * [AIOps explained](/en/topics/ai/what-is-aiops) * [What is parameter-efficient fine-tuning (PEFT)?](/en/topics/ai/what-is-peft) * [What is AI in healthcare?](/en/topics/ai/what-is-ai-in-healthcare) * [What is LLMops](/en/topics/ai/llmops) * [What is explainable AI?](/en/topics/ai/what-explainable-ai) * [What is machine learning?](/en/topics/ai/what-is-machine-learning) * [Agentic AI vs. generative AI](/en/topics/ai/agentic-ai-vs-generative-ai) * [What are foundation models for AI?](/en/topics/ai/what-are-foundation-models) * [What is deep learning?](/en/topics/ai/what-is-deep-learning) * [What is MLOps?](/en/topics/ai/what-is-mlops) * [What is sovereign AI?](/en/topics/ai/sovereign-ai) * [What is Model Context Protocol (MCP)?](/en/topics/ai/what-is-model-context-protocol-mcp) * [What is retrieval-augmented generation?](/en/topics/ai/what-is-retrieval-augmented-generation) * [What is distributed inference?](/en/topics/ai/what-is-distributed-inference) * [Understanding AI/ML use cases](/en/topics/ai/ai-ml-use-cases) * [What is agentic AI?](/en/topics/ai/what-is-agentic-ai) * [What is AI in the public sector?](/en/topics/ai/what-is-ai-in-the-public-sector) * [Understanding AI in telecommunications with Red Hat](/en/topics/ai/understanding-ai-in-telecommunications) * [What is Model-as-a-Service?](/en/topics/ai/what-is-models-as-a-service) * [What is an AI platform?](/en/topics/ai/what-is-an-ai-platform) * [What is AI security?](/en/topics/ai/what-is-ai-security) * [What is enterprise AI?](/en/topics/ai/what-is-enterprise-ai) * [What is vLLM?](/en/topics/ai/what-is-vllm) * [What is generative AI?](/en/topics/ai/what-is-generative-ai) * [What is AI inference?](/en/topics/ai/what-is-ai-inference) * [What is llm-d?](/en/topics/ai/what-is-llm-d) * [RAG vs. fine-tuning](/en/topics/ai/rag-vs-fine-tuning) * [What are large language models?](/en/topics/ai/what-are-large-language-models) * [What is edge AI?](/en/topics/edge-computing/what-is-edge-ai) * [What are Granite models?](/en/topics/ai/what-are-granite-models) * [What are intelligent applications?](/en/topics/ai/what-are-intelligent-applications) * [SLMs vs LLMs: What are small language models?](/en/topics/ai/llm-vs-slm) * [Predictive AI vs generative AI](/en/topics/ai/predictive-ai-vs-generative-ai) * [AI in banking](/en/topics/ai/ai-in-banking) * [vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama) * [How vLLM accelerates AI inference: 3 enterprise use cases](/en/topics/ai/how-vllm-accelerates-ai-inference-3-enterprise-use-cases) * [What is InstructLab?](/en/topics/ai/what-is-instructlab) * [Why choose Red Hat Ansible Automation Platform as your AI foundation?](/en/topics/automation/automation-and-ai) * [LoRA vs. QLoRA](/en/topics/ai/lora-vs-qlora) * [Edge solutions for real-time decision making](/en/topics/edge-computing/edge-solutions-real-time-decision-making) * [What are predictive analytics](/en/topics/automation/how-predictive-analytics-improve-it-performance) * [How Kubernetes can help AI/ML](/en/topics/cloud-computing/how-kubernetes-can-help-ai) * [What is Kubeflow?](/en/topics/cloud-computing/what-is-kubeflow) * [Accelerate MLOps with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/aiml) * [What is a data lake?](/en/topics/data-storage/what-is-a-data-lake) [More about this topic](/en/topics/ai "More about this topic")