From experiment to production: A reliable architecture for version-controlled MLOps

Many believe that building models is the most challenging thing in AI, but managing the data that powers those models is equally difficult. We’ve all been there: a model's performance shifts or a data analysis yields inconsistent results, and you’re left wondering, "Wait, which version of the dataset did I use for this training run or report?"

But, there’s good news. If you’re working in a Red Hat environment, our new AI quickstart helps address this exact problem. It combines the orchestration power of Red Hat OpenShift AI with the "Git-for-data" versioning capabilities of lakeFS.

The AI quickstart helps you build a cohesive ecosystem in which each layer solves a specific challenge in the AI lifecycle, providing a workflow where infrastructure, data management, and model development are smoothly integrated.

Here’s how this AI quickstart works and how you can get started.

The foundation: Red Hat OpenShift

At the base of everything is Red Hat OpenShift. While Kubernetes can often feel like a DIY project, Red Hat OpenShift provides a hardened, enterprise-grade foundation. It handles the heavy lifting of scalability and security so that when your AI models move from a local laptop to production, the underlying infrastructure can handle the load with ease.

The intelligence layer: Red Hat OpenShift AI

Sitting directly on top of that foundation is Red Hat OpenShift AI. This is where the actual work happens for data scientists. Rather than jumping between fragmented tools, this layer consolidates the entire environment—Jupyter Notebooks for experimentation, model serving for deployment, and automated pipelines—into a single, unified dashboard. It effectively bridges the gap between writing code and delivering a functional AI service.

The versioning engine: lakeFS

Data management is where lakeFS shines. In traditional development, we use Git to version code. lakeFS brings that same logic to the data itself. By acting as a data control plane over your object storage, it allows you to branch, commit, and revert data sets just as easily as you would a script.

Because lakeFS is built for multimodal data, it treats structured tables, semistructured JSON, and unstructured images or metadata with the same level of version control. This makes every AI model reproducible—if a model behaves unexpectedly, you can simply roll the data back to the exact state it was in when the model was trained.

What’s inside the AI quickstart?

The AI quickstart, Securing AI Fraud Models Through Advanced Data Versioning, more than a "hello world" demo—it’s a full lifecycle workflow based on a real-world fraud detection use case. When you run through it, you’ll learn how to:

• Train a model in isolation: Use lakeFS branches to create a sandbox for your data. You can experiment with new data preprocessing techniques without affecting the main production dataset branch.
• Version your artifacts: Every time you run a training pipeline, the AI quickstart shows you how to commit the specific state of your data. This enables 100% reproducibility if a model behaves strangely in the future—you can revert to the exact version of the data used to train it.
• Automate with pipelines: You can integrate lakeFS directly into OpenShift AI pipelines. The pipeline doesn't just run code, it creates a data snapshot at every step.
• Serve models efficiently: Deploy your trained fraud detection model using OpenShift AI’s single-model serving platform, with the model weights pulled directly from a versioned lakeFS repository.

Why this matters for machine learning operations (MLOps)

Most teams treat data as a "live" entity that is constantly changing, which can make auditing and debugging more challenging. However, by using lakeFS as an AI data control plane on Red Hat OpenShift, organizations gain powerful tools to help manage their data lifecycles more effectively.

A primary benefit of this setup is zero-copy branching, which allows teams to create a copy of massive datasets, such as a 1TB dataset for testing, in milliseconds, entirely without duplicating the underlying data. This capability naturally extends into data continuous integration and continuous delivery (CI/CD) practices, enabling teams to use "pre-merge hooks" that validate data quality before any information reaches the training pipeline. In the event that bad data ingestion ruins a model, the system provides instant rollbacks, letting users immediately revert the data repository back to its previous state.

This architecture is also designed to manage all AI data formats, including unstructured data and metadata. Teams can easily access and apply version control to any format, whether it's structured, semi-structured, or unstructured, alongside its associated metadata. Ultimately, this comprehensive control is vital for compliance. For highly regulated industries like financial services, healthcare, telecommunications (telco), and the public sector, this system provides a verifiable chain of custody for training data, so every model can be traced back to a specific, immutable dataset snapshot.

Business impact: Beyond technical reproducibility

While the technical advantages of this architecture are clear, its impact on the enterprise is also significant. By introducing lakeFS-based data versioning, ML teams can deliver up to 2-3 times the number of models with a smaller team. This system helps reduce environment drift, rework, and dataset duplication, so teams are able to increase their output without needing to hire more people.

This increased throughput is largely driven by faster experimentation cycles. Using zero-copy branching, teams can immediately test new datasets and features instead of waiting for large volumes of data to be duplicated or for infrastructure to be provisioned, significantly reducing testing time. Additionally, because logical branching avoids the physical replication of massive datasets, organizations can prevent unnecessary growth in cloud storage, helping to lower overall infrastructure costs.

Beyond speed and cost, the enterprise also can benefit from improved risk management and operational stability. Immutable data commits establish a verifiable chain of custody for training data, so that every deployed model can be traced back to a specific data snapshot. This level of compliance and audit readiness is crucial for meeting internal governance requirements and supporting highly regulated industries. Finally, if faulty data does manage to enter the pipelines, instant rollbacks minimize downtime, helping teams avoid operational disruptions and costly retraining cycles.

Get started

The benefit of this AI quickstart is its "ready-to-run" design, so non-experts can get started more easily. To begin, you'll need access to a Red Hat OpenShift cluster with OpenShift AI installed. While standard user access is sufficient for most tasks, you'll need cluster-admin permissions if you choose to configure an optional model registry. Deployment is straightforward, as the repository includes a makefile and automation scripts that handle the heavy lifting of deploying lakeFS, object storage, configuring the S3 gateway, and setting up the necessary data connections within your OpenShift AI project.

Once the prerequisites are in place, the workflow follows a simple sequence. First, you'll create a data science project directly within the OpenShift AI dashboard. Next, run the provided setup script to initialize your lakeFS repositories and object storage buckets. With the environment prepared, you can launch a workbench and clone the quickstart repository. Finally, by following the provided notebooks, you'll learn how to branch your data, train the model, and see how lakeFS tracks every change.

Ready to try it? Check out the AI quickstart and get started!