--- url: 'https://www.redhat.com/topics/cloud-computing/how-kubernetes-can-help-ai' title: 'How Kubernetes can help AI/ML' date: '2024-11-20T22:53:14+00:00' updated: '2026-09-04T18:37:30+00:00' type: website summary: 'Kubernetes can assist with AI/ML workloads by making code consistently reproducible, portable, and scalable across diverse environments.' tags: - Article - 'Red Hat OpenShift AI' - AI - 'Application services' - Cloud - 'Artificial intelligence' - Containers image: 'https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-rh-og-image.png' published: true schema: '@context': 'https://schema.org' '@graph': - '@type': Article '@id': 'https://www.redhat.com/en/topics/cloud-computing/how-kubernetes-can-help-ai' headline: 'How Kubernetes can help AI/ML' name: 'How Kubernetes can help AI/ML' about: 'Artificial Intelligence, Machine Learning, Kubernetes, MLOps' description: 'Discover how Kubernetes streamlines the AI/ML lifecycle by providing data science teams with scalability, portability, and resource efficiency to train, test, and deploy machine learning models.' datePublished: '2024-11-20T22:53:14+0000' isAccessibleForFree: 'True' author: '@type': Organization '@id': 'https://www.redhat.com' name: 'Red Hat' url: 'https://www.redhat.com' logo: '@type': ImageObject representativeOfPage: 'False' url: 'https://www.redhat.com/themes/custom/rh_pxt/logo.svg' publisher: '@type': Organization '@id': 'https://www.redhat.com/' name: 'Red Hat' url: 'https://www.redhat.com/' logo: '@type': ImageObject representativeOfPage: 'False' url: 'https://www.redhat.com/themes/custom/rh_pxt/logo.svg' mainEntityOfPage: 'https://www.redhat.com/en/topics/cloud-computing/how-kubernetes-can-help-ai' - '@type': FAQPage '@id': 'https://www.redhat.com/en/topics/cloud-computing/how-kubernetes-can-help-ai' mainEntity: - '@type': Question name: 'What is the role of containers in AI/ML development?' answerCount: '1' acceptedAnswer: '@type': Answer text: 'Containers package and isolate applications along with all necessary libraries and dependencies, providing lightweight portability across different platforms while separating developer and operations responsibilities.' author: Organization - '@type': Question name: 'What does Kubernetes bring to AI/ML workloads?' answerCount: '1' acceptedAnswer: '@type': Answer text: 'Kubernetes provides scalability to accommodate large-scale processing, efficiency by optimizing resource allocation and scheduling, portability across platforms without vendor lock-in, and fault tolerance with self-healing capabilities.' author: 'https://www.redhat.com/' - '@type': Question name: 'How do you deploy ML models on Kubernetes?' answerCount: '1' acceptedAnswer: '@type': Answer text: 'Organizations can deploy ML models on Kubernetes by using a Kubernetes architecture to automate portions of the ML lifecycle and implementing toolkits such as Kubeflow to streamline, serve, and orchestrate ML pipelines and MLOps at scale.' author: 'Red Hat' - '@type': Question name: 'How can Red Hat help with AI/ML?' answerCount: '1' acceptedAnswer: '@type': Answer text: 'Red Hat offers Red Hat AI and Red Hat OpenShift AI to provide a reliable, flexible, and scalable enterprise MLOps platform to build, train, deploy, and monitor predictive and generative AI models across hybrid cloud environments.' author: 'https://www.redhat.com/' --- 1. [Topics](/en/topics "Topics") 2. [Artificial intelligence](/en/topics/ai "Artificial intelligence") 3. [How Kubernetes can help AI/ML](# "How Kubernetes can help AI/ML") # How Kubernetes can help AI/ML Published June 24, 2026•*4*-minute read CopiedCopy failedCopy URL ## Jump to section OverviewContainers and AI/ML Kubernetes and AI/MLDeploying ML models on KubernetesHow Red Hat can helpSolution pattern ## Overview [Kubernetes](/en/topics/containers/what-is-kubernetes) can assist with AI/ML workloads by making code consistently reproducible, portable, and scalable across diverse environments. Kubernetes is an open source platform that automates Linux container operations by eliminating many of the manual processes involved in deploying and scaling containerized applications. Kubernetes is key to streamlining the AI/ML lifecycle because it provides data scientists the agility, flexibility, portability, and scalability to train, test, and deploy ML models. [Explore Red Hat AI ](/en/products/ai) ## The role of containers in AI/ML development When building machine learning enabled applications, the workflow is not linear, and the stages of research, development, and production are in perpetual motion as teams work to continuously integrate and continuously deliver ([CI/CD](/en/topics/devops/what-is-ci-cd)). The process of building, testing, merging, and deploying new data, algorithms, and versions of an application creates a lot of moving pieces, which can be difficult to manage. That’s where [containers](/en/topics/containers) come in. Containers are a [Linux ](/en/topics/linux)technology that allow you to package and isolate applications along with all the libraries and dependencies it needs to run. Containers don’t require an entire operating system, only the exact components it needs to operate, which makes it lightweight and portable. This provides an ease of deployment for operations and confidence for developers that their applications will run exactly the same way on different platforms or operating systems. Another benefit of containers is that they help reduce conflicts between your development and operations teams by separating areas of responsibility. And when developers can focus on their apps and operations teams can focus on the infrastructure, integrating new code into an application as it grows and evolves throughout its lifecycle becomes more seamless and efficient. [Start buidling your AI/ML environment](/en/resources/building-production-ready-ai-environment-ebook) ## Red Hat resources [Keep reading](/en/resources "Keep reading") ## What Kubernetes brings to AI/ML workloads **Scalability:** Kubernetes allows users to scale ML workloads up or down, depending on demand. This ensures that machine learning pipelines can accommodate large-scale processing and training without interfering with other elements of the project. **Efficiency:** Kubernetes optimizes resource allocation by scheduling workloads onto nodes based on their availability and capacity. By ensuring that computing resources are being utilized with intention, users can expect a reduction in cost and an increase in performance. **Portability:** Kubernetes provides a standardized, platform-agnostic environment that allows data scientists to develop one ML model and deploy it across multiple environments and cloud platforms. This means not having to worry about compatibility issues and vendor lock-in. **Fault tolerance:** With built-in fault tolerance and self-healing capabilities, users can trust Kubernetes to keep ML pipelines running even in the event of a hardware or software failure. ## Deploying ML models on Kubernetes The machine learning lifecycle is made up of many different elements that, if managed separately, would be time consuming and resource intensive to operate and maintain. With a Kubernetes architecture, organizations can automate portions of the ML lifecycle, removing the need for manual intervention and creating more efficiency. Toolkits such as [Kubeflow](/en/topics/cloud-computing/what-is-kubeflow) can be implemented to assist developers in streamlining and serving the trained ML workloads on Kubernetes. Kubeflow solves many of the challenges involved in orchestrating machine learning pipelines by providing a set of tools and APIs that simplify the process of training and deploying ML models at scale. Kubeflow also helps standardize and organize [machine learning operations (MLOps)](/en/topics/ai/what-is-mlops). [Learn how to operationalize Kubeflow on OpenShift](https://cloud.redhat.com/blog/operationalizing-kubeflow-in-openshift) ## How Red Hat can help Kubernetes can help you streamline AI/ML workloads, but you still need a platform to experiment, serve models, and deliver your applications. [Red Hat® AI](/en/products/ai) is our portfolio of AI products built on solutions our customers already trust. This foundation helps our products remain reliable, flexible, and scalable. Red Hat AI can help organizations: - Adopt and innovate with AI quickly. - Break down the complexities of delivering AI solutions. - Deploy anywhere. ### Stay flexible while you scale Red Hat AI includes [Red Hat OpenShift® AI](/en/products/ai/openshift-ai): an integrated MLOps platform that can manage the lifecycle of both [predictive and generative AI models](/en/topics/ai/predictive-ai-vs-generative-ai). This AI platform provides a space to build, train, deploy, and monitor AI/ML workloads in on-premise datacenters or closer to where data is located. This makes it easier to scale operations to the cloud or at the edge when needed. Layering Kubernetes with Red Hat AI will allow your team to stay nimble when delivering AI applications across hybrid cloud environments. [Read more about Red Hat OpenShift AI ](/en/products/ai/openshift-ai) ## Solution pattern: AI apps with Red Hat & NVIDIA AI Enterprise ### Create a RAG application [Red Hat OpenShift AI](/en/products/ai/openshift-ai "Red Hat OpenShift AI") is a platform for building data science projects and serving AI-enabled applications. You can integrate all the tools you need to support [retrieval-augmented generation (RAG)](/en/topics/ai/what-is-retrieval-augmented-generation "What is retrieval-augmented generation?"), a method for getting AI answers from your own reference documents. When you connect OpenShift AI with NVIDIA AI Enterprise, you can experiment with [large language models (LLMs)](/en/topics/ai/what-are-large-language-models "What are large language models?") to find the optimal model for your application. ### Build a pipeline for documents To make use of RAG, you first need to ingest your documents into a vector database. In our example app, we embed a set of product documents in a Redis database. Since these documents change frequently, we can create a pipeline for this process that we’ll run periodically, so we always have the latest versions of the documents. ### Browse the LLM catalog NVIDIA AI Enterprise gives you access to a catalog of different LLMs, so you can try different choices and select the model that delivers the best results. The models are hosted in the NVIDIA API catalog. Once you’ve set up an API token, you can deploy a model using the NVIDIA NIM model serving platform directly from OpenShift AI. ### Choose the right model As you test different LLMs, your users can rate each generated response. You can set up a Grafana monitoring dashboard to compare the ratings, as well as latency and response time for each model. Then you can use that data to choose the best LLM to use in production. [](/rhdc/managed-files/solutions-pattern-nvidia.jpg)[Download the PDF](/rhdc/managed-files/solution-pattern-RHOAI-NVIDIA-demo-dl.pdf) ### The official Red Hat blog Get the latest information about our ecosystem of customers, partners, and communities. [Keep reading](/en/blog "The official Red Hat blog") ### All Red Hat product trials Our no-cost product trials help you gain hands-on experience, prepare for a certification, or assess if a product is right for your organization. [Keep reading](/en/products/trials "All Red Hat product trials") ## Keep reading ### What is MLflow? MLflow is a tool that helps developers keep machine learning (ML) and AI projects organized from early experimentation to live deployment. [Read the article](/en/topics/ai/what-is-mlflow "What is MLflow?") ### MCP vs. APIs: What's the difference? MCP vs. APIs explained: Learn how each connects systems, supports AI workflows, and which is best for your application needs. [Read the article](/en/topics/ai/mcp-vs-apis "MCP vs. APIs") ### Understanding agentic AI use cases Explore real-world agentic AI use cases and see how these systems plan, decide, and act independently to overcome business challenges. [Read the article](/en/topics/ai/agentic-ai-use-cases "Understanding agentic AI use cases") ### Artificial intelligence resources ### Featured product - #### [Red Hat OpenShift AI](/en/products/ai/openshift-ai) An artificial intelligence (AI) platform that provides tools to rapidly develop, train, serve, and monitor models and AI-enabled applications. [See all products](/en/products "See all products") ### Related content - Analyst material [Podman for DevOps](/en/engage/packt-podman-for-devops-analyst-material) - Blog post [MLOps for edge AI: Preparing AI models for deployment at the edge](/en/blog/mlops-edge-ai-preparing-ai-models-deployment-edge) - Blog post [Reimagining the enterprise innovation engine in the agentic era](/en/blog/reimagining-enterprise-innovation-engine-agentic-era) - E-book [The enterprise inference playbook](/en/resources/enterprise-inference-playbook) ### Related articles - [What is MLflow?](/en/topics/ai/what-is-mlflow) - [MCP vs. APIs: What's the difference?](/en/topics/ai/mcp-vs-apis) - [Understanding agentic AI use cases](/en/topics/ai/agentic-ai-use-cases) - [What are predictive analytics](/en/topics/automation/how-predictive-analytics-improve-it-performance) - [AIOps explained](/en/topics/ai/what-is-aiops) - [What is PyTorch?](/en/topics/ai/what-is-pytorch) - [What is a Kubernetes operator?](/en/topics/containers/what-is-a-kubernetes-operator) - [What is a golden image?](/en/topics/linux/what-is-a-golden-image) - [What is container orchestration?](/en/topics/containers/what-is-container-orchestration) - [Introduction to Kubernetes architecture](/en/topics/containers/kubernetes-architecture) - [What is a Kubernetes cluster?](/en/topics/containers/what-is-a-kubernetes-cluster) - [Stateful vs stateless applications](/en/topics/cloud-native-apps/stateful-vs-stateless) - [What is Kubernetes?](/en/topics/containers/what-is-kubernetes) - [What is Docling?](/en/topics/ai/docling) - [Containers vs. VMs: Why not both?](/en/topics/containers/containers-vs-vms) - [What are Red Hat OpenShift Operators?](/en/technologies/cloud-computing/openshift/what-are-openshift-operators) - [What is network-attached storage?](/en/topics/data-storage/network-attached-storage) - [What is agentic AI?](/en/topics/ai/what-is-agentic-ai) - [Predictive AI vs generative AI](/en/topics/ai/predictive-ai-vs-generative-ai) - [What is Mixture of Experts (MoE)?](/en/topics/ai/mixture-of-experts) - [How vLLM accelerates AI inference: 3 enterprise use cases](/en/topics/ai/how-vllm-accelerates-ai-inference-3-enterprise-use-cases) - [What are large language models?](/en/topics/ai/what-are-large-language-models) - [RAG vs. fine-tuning](/en/topics/ai/rag-vs-fine-tuning) - [What is AI inference?](/en/topics/ai/what-is-ai-inference) - [SLMs vs LLMs: What are small language models?](/en/topics/ai/llm-vs-slm) - [What is generative AI?](/en/topics/ai/what-is-generative-ai) - [What is vLLM?](/en/topics/ai/what-is-vllm) - [vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama) - [What are Granite models?](/en/topics/ai/what-are-granite-models) - [What are intelligent applications?](/en/topics/ai/what-are-intelligent-applications) - [LoRA vs. QLoRA](/en/topics/ai/lora-vs-qlora) - [What is AgentOps?](/en/topics/ai/agentops) - [What is parameter-efficient fine-tuning (PEFT)?](/en/topics/ai/what-is-peft) - [What is Model Context Protocol (MCP)?](/en/topics/ai/what-is-model-context-protocol-mcp) - [What is distributed inference?](/en/topics/ai/what-is-distributed-inference) - [What are foundation models for AI?](/en/topics/ai/what-are-foundation-models) - [Understanding AI in telecommunications with Red Hat](/en/topics/ai/understanding-ai-in-telecommunications) - [What is LLMops](/en/topics/ai/llmops) - [What is machine learning?](/en/topics/ai/what-is-machine-learning) - [What is MLOps?](/en/topics/ai/what-is-mlops) - [What is AI in the public sector?](/en/topics/ai/what-is-ai-in-the-public-sector) - [What is explainable AI?](/en/topics/ai/what-explainable-ai) - [Agentic AI vs. generative AI](/en/topics/ai/agentic-ai-vs-generative-ai) - [What is retrieval-augmented generation?](/en/topics/ai/what-is-retrieval-augmented-generation) - [AI infrastructure explained](/en/topics/ai/ai-infrastructure-explained) - [What is AI in healthcare?](/en/topics/ai/what-is-ai-in-healthcare) - [What is deep learning?](/en/topics/ai/what-is-deep-learning) - [What is sovereign AI?](/en/topics/ai/sovereign-ai) - [What is enterprise AI?](/en/topics/ai/what-is-enterprise-ai) - [What is an AI platform?](/en/topics/ai/what-is-an-ai-platform) - [What is Model-as-a-Service?](/en/topics/ai/what-is-models-as-a-service) - [What is AI security?](/en/topics/ai/what-is-ai-security) - [What is llm-d?](/en/topics/ai/what-is-llm-d) - [What is edge AI?](/en/topics/edge-computing/what-is-edge-ai) - [What is an image builder?](/en/topics/linux/what-is-an-image-builder) - [Red Hat OpenShift for developers](/en/technologies/cloud-computing/openshift/developers) - [AI in banking](/en/topics/ai/ai-in-banking) - [What is a Linux container?](/en/topics/containers/whats-a-linux-container) - [Why choose Red Hat for Kubernetes?](/en/topics/containers/why-choose-red-hat-kubernetes) - [What is CaaS?](/en/topics/cloud-computing/what-is-caas) - [What is Podman?](/en/topics/containers/what-is-podman) - [What is Podman Desktop?](/en/topics/containers/what-is-podman-desktop) - [Edge computing with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/edge-computing) - [What is InstructLab?](/en/topics/ai/what-is-instructlab) - [Why choose Red Hat Ansible Automation Platform as your AI foundation?](/en/topics/automation/automation-and-ai) - [Red Hat OpenShift on VMware](/en/technologies/cloud-computing/openshift/vmware) - [What is KubeVirt?](/en/topics/virtualization/what-is-kubevirt) - [Why use Red Hat Ansible Automation Platform with Red Hat OpenShift?](/en/technologies/cloud-computing/openshift/ansible-on-openshift) - [Edge solutions for real-time decision making](/en/topics/edge-computing/edge-solutions-real-time-decision-making) - [What is the Kubernetes Java client?](/en/topics/cloud-computing/what-is-kubernetes-java-client) - [What are hosted control planes?](/en/topics/containers/what-are-hosted-control-planes) - [What is kubernetes security?](/en/topics/containers/kubernetes-security) - [What is Kubeflow?](/en/topics/cloud-computing/what-is-kubeflow) - [What are microservices?](/en/topics/microservices/what-are-microservices) - [OpenShift vs. OpenStack: What are the differences?](/en/technologies/cloud-computing/openshift/openshift-vs-openstack) - [What is container security?](/en/topics/security/container-security) - [What is Buildah?](/en/topics/containers/what-is-buildah) - [What are sandboxed containers](/en/topics/containers/sandboxed-containers) - [Accelerate MLOps with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/aiml) - [Kubernetes vs OpenStack](/en/topics/openstack/kubernetes-vs-openstack) - [What are validated patterns?](/en/topics/cloud-computing/what-are-validated-patterns) - [Kubernetes on AWS: Self-Managed vs. Managed Applications Platforms](/en/topics/containers/kubernetes-on-aws) - [Red Hat OpenShift vs. OKD](/en/topics/containers/red-hat-openshift-okd) - [Red Hat OpenShift vs. Kubernetes: What's the difference?](/en/technologies/cloud-computing/openshift/red-hat-openshift-kubernetes) - [What is high availability and disaster recovery for containers?](/en/topics/containers/high-availability-containers) - [Why run Apache Kafka on Kubernetes?](/en/topics/integration/why-run-apache-kafka-on-kubernetes) - [Spring on Kubernetes with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/spring) - [What are Red Hat OpenShift cloud services?](/en/technologies/cloud-computing/openshift/what-are-red-hat-openshift-cloud-services) - [VNF and CNF, what’s the difference?](/en/topics/cloud-native-apps/vnf-and-cnf-whats-the-difference) - [What is a container registry?](/en/topics/cloud-native-apps/what-is-a-container-registry) - [What is Skopeo?](/en/topics/containers/what-is-skopeo) - [Using Helm with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/helm) - [Kubernetes security best practices](/en/topics/containers/kubernetes-security-best-practices) - [Orchestrating Windows containers on Red Hat OpenShift](/en/technologies/cloud-computing/openshift/windows-containers-on-red-hat-openshift) - [High performance computing with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/high-performance-computing) - [Advantages of Kubernetes-native security](/en/topics/containers/advantages-of-kubernetes-native-security) - [What is KubeLinter?](/en/topics/containers/what-is-kubelinter) - [Container and Kubernetes compliance considerations](/en/topics/containers/compliance) - [Intro to Kubernetes security](/en/topics/containers/intro-kubernetes-security) - [How microservices support IT integration in healthcare](/en/topics/microservices/microservices-in-healthcare) - [Kubernetes cluster management](/en/technologies/cloud-computing/openshift/kubernetes-cluster-management) - [Red Hat OpenShift on IBM IT infrastructure](/en/technologies/cloud-computing/openshift/what-is-red-hat-openshift-on-IBM-IT-infrastructure) - [Red Hat OpenShift for business leaders](/en/technologies/cloud-computing/openshift/business-leaders) - [Cost management for Kubernetes on Red Hat OpenShift](/en/technologies/cloud-computing/openshift/cost-management) - [How to deploy Red Hat OpenShift](/en/technologies/cloud-computing/openshift/deploy-red-hat-openshift) - [What makes Red Hat OpenShift the right choice for SAP?](/en/technologies/cloud-computing/openshift/sap) - [Kubernetes-native Java development with Quarkus](/en/technologies/cloud-computing/openshift/quarkus) - [What is enterprise Kubernetes?](/en/topics/containers/what-is-enterprise-kubernetes) - [What makes Red Hat OpenShift the right choice for IT operations?](/en/technologies/cloud-computing/openshift/it-operations) - [What is Kubernetes role-based access control (RBAC)](/en/topics/containers/what-kubernetes-role-based-access-control-rbac) - [What is containerization?](/en/topics/cloud-native-apps/what-is-containerization) - [What was CoreOS and CoreOS container Linux](/en/technologies/cloud-computing/openshift/what-was-coreos) - [Learning Kubernetes basics](/en/topics/containers/learning-kubernetes-tutorial) - [What is service-oriented architecture?](/en/topics/cloud-native-apps/what-is-service-oriented-architecture) - [What is the Kubernetes API?](/en/topics/containers/what-is-the-kubernetes-api) - [What is Kubernetes cluster management?](/en/topics/containers/what-is-kubernetes-cluster-management) - [What is a Kubernetes deployment?](/en/topics/containers/what-is-kubernetes-deployment) - [Why choose the Red Hat build of Quarkus?](/en/topics/cloud-native-apps/why-choose-red-hat-quarkus) - [Introduction to Kubernetes patterns](/en/topics/cloud-native-apps/introduction-to-kubernetes-patterns) - [What is Quarkus?](/en/topics/cloud-native-apps/what-is-quarkus) - [What is Jaeger?](/en/topics/microservices/what-is-jaeger) - [What is a data lake?](/en/topics/data-storage/what-is-a-data-lake) - [What is Clair?](/en/topics/containers/what-is-clair) - [What is Knative?](/en/topics/microservices/what-is-knative) - [What is etcd?](/en/topics/containers/what-is-etcd) - [What is container-native virtualization?](/en/topics/containers/what-is-container-native-virtualization) - [Why choose Red Hat for microservices?](/en/topics/microservices/why-choose-red-hat-microservices) - [What is software-defined storage?](/en/topics/data-storage/software-defined-storage) - [Why choose Red Hat for containers?](/en/topics/containers/why-choose-red-hat-containers) - [What is Docker?](/en/topics/containers/what-is-docker) - [What is a Kubernetes pod?](/en/topics/containers/what-is-kubernetes-pod) [More about this topic](/en/topics/ai "More about this topic")