* [Topics](/en/topics "Topics") * [Artificial intelligence](/en/topics/ai "Artificial intelligence") * How Kubernetes can help AI/ML How Kubernetes can help AI/ML ============================= Published August 9, 2023•*4*-minute read Copy URL Jump to section --------------- OverviewContainers and AI/ML Kubernetes and AI/MLDeploying ML models on KubernetesHow Red Hat can helpSolution pattern Overview -------- [Kubernetes](/en/topics/containers/what-is-kubernetes) can assist with AI/ML workloads by making code consistently reproducible, portable, and scalable across diverse environments. [Explore Red Hat AI](/en/products/ai) The role of containers in AI/ML development ------------------------------------------- When building machine learning enabled applications, the workflow is not linear, and the stages of research, development, and production are in perpetual motion as teams work to continuously integrate and continuously deliver ([CI/CD](/en/topics/devops/what-is-ci-cd)). The process of building, testing, merging, and deploying new data, algorithms, and versions of an application creates a lot of moving pieces, which can be difficult to manage. That’s where [containers](/en/topics/containers) come in. Containers are a [Linux](/en/topics/linux) technology that allow you to package and isolate applications along with all the libraries and dependencies it needs to run. Containers don’t require an entire operating system, only the exact components it needs to operate, which makes it lightweight and portable. This provides an ease of deployment for operations and confidence for developers that their applications will run exactly the same way on different platforms or operating systems. Another benefit of containers is that they help reduce conflicts between your development and operations teams by separating areas of responsibility. And when developers can focus on their apps and operations teams can focus on the infrastructure, integrating new code into an application as it grows and evolves throughout its lifecycle becomes more seamless and efficient. [Start buidling your AI/ML environment](/en/resources/building-production-ready-ai-environment-ebook) Red Hat resources ----------------- [Keep reading](/en/resources "Keep reading") What Kubernetes brings to AI/ML workloads ----------------------------------------- Kubernetes is an open source platform that automates Linux container operations by eliminating many of the manual processes involved in deploying and scaling containerized applications. Kubernetes is key to streamlining the machine learning lifecycle as it provides data scientists the agility, flexibility, portability, and scalability to train, test, and deploy ML models. **Scalability:** Kubernetes allows users to scale ML workloads up or down, depending on demand. This ensures that machine learning pipelines can accommodate large-scale processing and training without interfering with other elements of the project. **Efficiency:** Kubernetes optimizes resource allocation by scheduling workloads onto nodes based on their availability and capacity. By ensuring that computing resources are being utilized with intention, users can expect a reduction in cost and an increase in performance. **Portability:** Kubernetes provides a standardized, platform-agnostic environment that allows data scientists to develop one ML model and deploy it across multiple environments and cloud platforms. This means not having to worry about compatibility issues and vendor lock-in. **Fault tolerance:** With built-in fault tolerance and self-healing capabilities, users can trust Kubernetes to keep ML pipelines running even in the event of a hardware or software failure. Deploying ML models on Kubernetes --------------------------------- The machine learning lifecycle is made up of many different elements that, if managed separately, would be time consuming and resource intensive to operate and maintain. With a Kubernetes architecture, organizations can automate portions of the ML lifecycle, removing the need for manual intervention and creating more efficiency. Toolkits such as [Kubeflow](/en/topics/cloud-computing/what-is-kubeflow) can be implemented to assist developers in streamlining and serving the trained ML workloads on Kubernetes. Kubeflow solves many of the challenges involved in orchestrating machine learning pipelines by providing a set of tools and APIs that simplify the process of training and deploying ML models at scale. Kubeflow also helps standardize and organize [machine learning operations (MLOps)](/en/topics/ai/what-is-mlops). [Learn how to operationalize Kubeflow on OpenShift](https://cloud.redhat.com/blog/operationalizing-kubeflow-in-openshift) How Red Hat can help -------------------- Kubernetes can help you streamline AI/ML workloads, but you still need a platform to experiment, serve models, and deliver your applications. [Red Hat® AI](/en/products/ai) is our portfolio of AI products built on solutions our customers already trust. This foundation helps our products remain reliable, flexible, and scalable. Red Hat AI can help organizations: * Adopt and innovate with AI quickly. * Break down the complexities of delivering AI solutions. * Deploy anywhere. [Explore Red Hat AI](/en/products/ai) ### Stay flexible while you scale Red Hat AI includes [Red Hat OpenShift AI](/en/products/ai/openshift-ai): an integrated MLOps platform that can manage the lifecycle of both [predictive and generative AI models](/en/topics/ai/predictive-ai-vs-generative-ai). This AI platform provides a space to build, train, deploy, and monitor AI/ML workloads in on-premise datacenters or closer to where data is located. This makes it easier to scale operations to the cloud or at the edge when needed. Layering kubernetes with Red Hat AI will allow your team to stay nimble when delivering AI applications across hybrid cloud environments. [Read more about Red Hat OpenShift AI](/en/products/ai/openshift-ai) Solution pattern: AI apps with Red Hat & NVIDIA AI Enterprise ------------------------------------------------------------- ### Create a RAG application [Red Hat OpenShift AI](/en/products/ai/openshift-ai "Red Hat OpenShift AI") is a platform for building data science projects and serving AI-enabled applications. You can integrate all the tools you need to support [retrieval-augmented generation (RAG)](/en/topics/ai/what-is-retrieval-augmented-generation "What is retrieval-augmented generation?"), a method for getting AI answers from your own reference documents. When you connect OpenShift AI with NVIDIA AI Enterprise, you can experiment with [large language models (LLMs)](/en/topics/ai/what-are-large-language-models "What are large language models?") to find the optimal model for your application. ### Build a pipeline for documents To make use of RAG, you first need to ingest your documents into a vector database. In our example app, we embed a set of product documents in a Redis database. Since these documents change frequently, we can create a pipeline for this process that we’ll run periodically, so we always have the latest versions of the documents. ### Browse the LLM catalog NVIDIA AI Enterprise gives you access to a catalog of different LLMs, so you can try different choices and select the model that delivers the best results. The models are hosted in the NVIDIA API catalog. Once you’ve set up an API token, you can deploy a model using the NVIDIA NIM model serving platform directly from OpenShift AI. ### Choose the right model As you test different LLMs, your users can rate each generated response. You can set up a Grafana monitoring dashboard to compare the ratings, as well as latency and response time for each model. Then you can use that data to choose the best LLM to use in production. [![An architecture diagram shows an application built using Red Hat OpenShift AI and NVIDIA AI Enterprise. Components include OpenShift GitOps for connecting to GitHub and handling DevOps interactions, Grafana for monitoring, OpenShift AI for data science, Redis as a vector database, and Quay as an image registry. These components all flow to the app frontend and backend. These components are built on Red Hat OpenShift AI, with an integration with ai.nvidia.com.](/rhdc/managed-files/solutions-pattern-nvidia.jpg "An architecture diagram shows an application built using Red Hat OpenShift AI and NVIDIA AI Enterprise. Components include OpenShift GitOps for connecting to GitHub and handling DevOps interactions, Grafana for monitoring, OpenShift AI for data science, Redis as a vector database, and Quay as an image registry. These components all flow to the app frontend and backend. These components are built on Red Hat OpenShift AI, with an integration with ai.nvidia.com.")](/rhdc/managed-files/solutions-pattern-nvidia.jpg) [Download the PDF](/rhdc/managed-files/solution-pattern-RHOAI-NVIDIA-demo-dl.pdf) The official Red Hat blog ------------------------- Get the latest information about our ecosystem of customers, partners, and communities. [Keep reading](/en/blog "The official Red Hat blog") All Red Hat product trials -------------------------- Our no-cost product trials help you gain hands-on experience, prepare for a certification, or assess if a product is right for your organization. [Keep reading](/en/products/trials "All Red Hat product trials") Keep reading ------------ ### What is vLLM? vLLM is a collection of open source code that helps language models perform calculations more efficiently. [Read the article](/en/topics/ai/what-is-vllm "What is vLLM?") ### What is generative AI? Generative AI is a kind of artificial intelligence technology that relies on deep learning models trained on large data sets to create new content. [Read the article](/en/topics/ai/what-is-generative-ai "article | what is generative ai?") ### AIOps explained AIOps (AI for IT operations) is an approach to automating IT operations with machine learning and other advanced AI techniques. [Read the article](/en/topics/ai/what-is-aiops "article | AIOps explained") Artificial intelligence resources --------------------------------- ### Featured product * #### [Red Hat OpenShift AI](/en/products/ai/openshift-ai) An artificial intelligence (AI) platform that provides tools to rapidly develop, train, serve, and monitor models and AI-enabled applications. [See all products](/en/technologies/all-products "See all products") ### Related content * Blog post [What does “AI security” mean and why does it matter to your business?](/en/blog/what-does-ai-security-mean-and-why-does-it-matter-your-business) * Case study [[node:rh-smart-meta-title]](/en/resources/holcim-digital-center-case-study) * Blog post [From experiment to production: A reliable architecture for version-controlled MLOps](/en/blog/experiment-production-reliable-architecture-version-controlled-mlops) * Blog post [The state of cloud-native security 2026: Maturity gaps and the automation mandate](/en/blog/state-cloud-native-security-2026-maturity-gaps-and-automation-mandate) ### Related articles * [What is vLLM?](/en/topics/ai/what-is-vllm) * [What is generative AI?](/en/topics/ai/what-is-generative-ai) * [AI infrastructure explained](/en/topics/ai/ai-infrastructure-explained) * [AIOps explained](/en/topics/ai/what-is-aiops) * [What is Model Context Protocol (MCP)?](/en/topics/ai/what-is-model-context-protocol-mcp) * [What is AI inference?](/en/topics/ai/what-is-ai-inference) * [Agentic AI vs. generative AI](/en/topics/ai/agentic-ai-vs-generative-ai) * [What is AI security?](/en/topics/ai/what-is-ai-security) * [What is distributed inference?](/en/topics/ai/what-is-distributed-inference) * [What is enterprise AI?](/en/topics/ai/what-is-enterprise-ai) * [What is agentic AI?](/en/topics/ai/what-is-agentic-ai) * [What is llm-d?](/en/topics/ai/what-is-llm-d) * [What is an AI platform?](/en/topics/ai/what-is-an-ai-platform) * [What are large language models?](/en/topics/ai/what-are-large-language-models) * [RAG vs. fine-tuning](/en/topics/ai/rag-vs-fine-tuning) * [What is edge AI?](/en/topics/edge-computing/what-is-edge-ai) * [What is retrieval-augmented generation?](/en/topics/ai/what-is-retrieval-augmented-generation) * [What is MLOps?](/en/topics/ai/what-is-mlops) * [What is an image builder?](/en/topics/linux/what-is-an-image-builder) * [Red Hat OpenShift for developers](/en/technologies/cloud-computing/openshift/developers) * [What are Granite models?](/en/topics/ai/what-are-granite-models) * [What are intelligent applications?](/en/topics/ai/what-are-intelligent-applications) * [Understanding AI in telecommunications with Red Hat](/en/topics/ai/understanding-ai-in-telecommunications) * [What is LLMops](/en/topics/ai/llmops) * [What is AI in the public sector?](/en/topics/ai/what-is-ai-in-the-public-sector) * [What is Models-as-a-Service?](/en/topics/ai/what-is-models-as-a-service) * [SLMs vs LLMs: What are small language models?](/en/topics/ai/llm-vs-slm) * [What is AI in healthcare?](/en/topics/ai/what-is-ai-in-healthcare) * [What is a Linux container?](/en/topics/containers/whats-a-linux-container) * [AI in banking](/en/topics/ai/ai-in-banking) * [Predictive AI vs generative AI](/en/topics/ai/predictive-ai-vs-generative-ai) * [What is deep learning?](/en/topics/ai/what-is-deep-learning) * [Why choose Red Hat for Kubernetes?](/en/topics/containers/why-choose-red-hat-kubernetes) * [What are foundation models for AI?](/en/topics/ai/what-are-foundation-models) * [What is CaaS?](/en/topics/cloud-computing/what-is-caas) * [What is Podman?](/en/topics/containers/what-is-podman) * [What is Podman Desktop?](/en/topics/containers/what-is-podman-desktop) * [vLLM vs. Ollama: When to use each framework](/en/topics/ai/vllm-vs-ollama) * [What is explainable AI?](/en/topics/ai/what-explainable-ai) * [Containers vs. VMs: Similarities, differences, and combined approaches](/en/topics/containers/containers-vs-vms) * [How vLLM accelerates AI inference: 3 enterprise use cases](/en/topics/ai/how-vllm-accelerates-ai-inference-3-enterprise-use-cases) * [What is machine learning?](/en/topics/ai/what-is-machine-learning) * [Edge computing with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/edge-computing) * [What is InstructLab?](/en/topics/ai/what-is-instructlab) * [What is container orchestration?](/en/topics/containers/what-is-container-orchestration) * [What is parameter-efficient fine-tuning (PEFT)?](/en/topics/ai/what-is-peft) * [Why choose Red Hat Ansible Automation Platform as your AI foundation?](/en/topics/automation/automation-and-ai) * [LoRA vs. QLoRA](/en/topics/ai/lora-vs-qlora) * [Stateful vs stateless applications](/en/topics/cloud-native-apps/stateful-vs-stateless) * [What is Kubernetes?](/en/topics/containers/what-is-kubernetes) * [Red Hat OpenShift on VMware](/en/technologies/cloud-computing/openshift/vmware) * [What is KubeVirt?](/en/topics/virtualization/what-is-kubevirt) * [Why use Red Hat Ansible Automation Platform with Red Hat OpenShift?](/en/technologies/cloud-computing/openshift/ansible-on-openshift) * [Edge solutions for real-time decision making](/en/topics/edge-computing/edge-solutions-real-time-decision-making) * [What is the Kubernetes Java client?](/en/topics/cloud-computing/what-is-kubernetes-java-client) * [What are hosted control planes?](/en/topics/containers/what-are-hosted-control-planes) * [What is kubernetes security?](/en/topics/containers/kubernetes-security) * [What are predictive analytics](/en/topics/automation/how-predictive-analytics-improve-it-performance) * [Understanding AI/ML use cases](/en/topics/ai/ai-ml-use-cases) * [What is Kubeflow?](/en/topics/cloud-computing/what-is-kubeflow) * [What are microservices?](/en/topics/microservices/what-are-microservices) * [What is container security?](/en/topics/security/container-security) * [OpenShift vs. OpenStack: What are the differences?](/en/technologies/cloud-computing/openshift/openshift-vs-openstack) * [What are sandboxed containers](/en/topics/containers/sandboxed-containers) * [What is Buildah?](/en/topics/containers/what-is-buildah) * [Accelerate MLOps with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/aiml) * [Kubernetes vs OpenStack](/en/topics/openstack/kubernetes-vs-openstack) * [What are validated patterns?](/en/topics/cloud-computing/what-are-validated-patterns) * [Kubernetes on AWS: Self-Managed vs. Managed Applications Platforms](/en/topics/containers/kubernetes-on-aws) * [Red Hat OpenShift vs. OKD](/en/topics/containers/red-hat-openshift-okd) * [Red Hat OpenShift vs. Kubernetes: What's the difference?](/en/technologies/cloud-computing/openshift/red-hat-openshift-kubernetes) * [What is high availability and disaster recovery for containers?](/en/topics/containers/high-availability-containers) * [Why run Apache Kafka on Kubernetes?](/en/topics/integration/why-run-apache-kafka-on-kubernetes) * [Spring on Kubernetes with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/spring) * [What is a golden image?](/en/topics/linux/what-is-a-golden-image) * [What are Red Hat OpenShift cloud services?](/en/technologies/cloud-computing/openshift/what-are-red-hat-openshift-cloud-services) * [VNF and CNF, what’s the difference?](/en/topics/cloud-native-apps/vnf-and-cnf-whats-the-difference) * [What is a container registry?](/en/topics/cloud-native-apps/what-is-a-container-registry) * [What is Skopeo?](/en/topics/containers/what-is-skopeo) * [What are Red Hat OpenShift Operators?](/en/technologies/cloud-computing/openshift/what-are-openshift-operators) * [Using Helm with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/helm) * [Kubernetes security best practices](/en/topics/containers/kubernetes-security-best-practices) * [Orchestrating Windows containers on Red Hat OpenShift](/en/technologies/cloud-computing/openshift/windows-containers-on-red-hat-openshift) * [What is a Kubernetes operator?](/en/topics/containers/what-is-a-kubernetes-operator) * [High performance computing with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/high-performance-computing) * [Advantages of Kubernetes-native security](/en/topics/containers/advantages-of-kubernetes-native-security) * [What is KubeLinter?](/en/topics/containers/what-is-kubelinter) * [Intro to Kubernetes security](/en/topics/containers/intro-kubernetes-security) * [Container and Kubernetes compliance considerations](/en/topics/containers/compliance) * [How microservices support IT integration in healthcare](/en/topics/microservices/microservices-in-healthcare) * [Kubernetes cluster management](/en/technologies/cloud-computing/openshift/kubernetes-cluster-management) * [Red Hat OpenShift on IBM IT infrastructure](/en/technologies/cloud-computing/openshift/what-is-red-hat-openshift-on-IBM-IT-infrastructure) * [Red Hat OpenShift for business leaders](/en/technologies/cloud-computing/openshift/business-leaders) * [How to deploy Red Hat OpenShift](/en/technologies/cloud-computing/openshift/deploy-red-hat-openshift) * [Cost management for Kubernetes on Red Hat OpenShift](/en/technologies/cloud-computing/openshift/cost-management) * [What is enterprise Kubernetes?](/en/topics/containers/what-is-enterprise-kubernetes) * [What makes Red Hat OpenShift the right choice for IT operations?](/en/technologies/cloud-computing/openshift/it-operations) * [What makes Red Hat OpenShift the right choice for SAP?](/en/technologies/cloud-computing/openshift/sap) * [Kubernetes-native Java development with Quarkus](/en/technologies/cloud-computing/openshift/quarkus) * [What is Kubernetes role-based access control (RBAC)](/en/topics/containers/what-kubernetes-role-based-access-control-rbac) * [What is containerization?](/en/topics/cloud-native-apps/what-is-containerization) * [What was CoreOS and CoreOS container Linux](/en/technologies/cloud-computing/openshift/what-was-coreos) * [Learning Kubernetes basics](/en/topics/containers/learning-kubernetes-tutorial) * [What is service-oriented architecture?](/en/topics/cloud-native-apps/what-is-service-oriented-architecture) * [What is the Kubernetes API?](/en/topics/containers/what-is-the-kubernetes-api) * [What is Kubernetes cluster management?](/en/topics/containers/what-is-kubernetes-cluster-management) * [What is a Kubernetes deployment?](/en/topics/containers/what-is-kubernetes-deployment) * [Why choose the Red Hat build of Quarkus?](/en/topics/cloud-native-apps/why-choose-red-hat-quarkus) * [Introduction to Kubernetes architecture](/en/topics/containers/kubernetes-architecture) * [Introduction to Kubernetes patterns](/en/topics/cloud-native-apps/introduction-to-kubernetes-patterns) * [What is a Kubernetes cluster?](/en/topics/containers/what-is-a-kubernetes-cluster) * [What is Quarkus?](/en/topics/cloud-native-apps/what-is-quarkus) * [What is Jaeger?](/en/topics/microservices/what-is-jaeger) * [What is a data lake?](/en/topics/data-storage/what-is-a-data-lake) * [What is etcd?](/en/topics/containers/what-is-etcd) * [What is container-native virtualization?](/en/topics/containers/what-is-container-native-virtualization) * [What is Clair?](/en/topics/containers/what-is-clair) * [What is Knative?](/en/topics/microservices/what-is-knative) * [Why choose Red Hat for microservices?](/en/topics/microservices/why-choose-red-hat-microservices) * [Why choose Red Hat for containers?](/en/topics/containers/why-choose-red-hat-containers) * [What is Docker?](/en/topics/containers/what-is-docker) * [What is a Kubernetes pod?](/en/topics/containers/what-is-kubernetes-pod) [More about this topic](/en/topics/ai "More about this topic")