From fragmented to flawless: Unifying the AI development lifecycle

AI teams struggle because data, experiments, models, and deployments often live in separate systems. The DagsHub AI quickstart for Red Hat OpenShift AI gives teams a way to manage dataset versioning, annotation, experiment tracking, model registry, and deployment workflows in a single OpenShift-based environment.

Developing robust AI solutions demands managing a complex ecosystem of data, experiments, and models. One of the primary hurdles data science teams face is the fragmentation of their initial workflows. To build effective and accurate models, teams must be able to seamlessly connect multiple data sources to enrich, query, visualize, and annotate datasets. When data operations are disconnected, managing and preparing high-quality data can become a manual, error-prone bottleneck.

The challenges don't stop at data preparation. As teams iterate on their algorithms, they often struggle with the lack of unified tools for tracking experiment progress, understanding trends, and comparing results across numerous training runs. Without a simplified and unified way to track these changes, reproducibility is nearly impossible. Furthermore, as models move from the laboratory toward production, organizations face the critical, highly complex task of managing model versions and deployment, as well as tracing the lineage from the final model back to its original source data. Overcoming these fragmented, disconnected workflows is essential for achieving end-to-end traceability and scaling reliable AI development.

Fortunately, there's a solution for teams operating within a Red Hat environment. Our AI quickstart with DagsHub directly addresses development bottlenecks by combining the orchestration of Red Hat OpenShift AI with DagsHub's ability to version, curate, and annotate data, manage models, and track experiments, alongside code versions within a single, unified repository. It provides an intuitive experience where the underlying infrastructure, dataset management, experiment tracking, and model creation work together in a single OpenShift-based workflow.

The underlying layer: Red Hat OpenShift

Red Hat OpenShift acts as the foundation for this architecture. Rather than dealing with the manual, build-it-yourself setup often associated with standard Kubernetes, OpenShift supplies an enterprise-ready environment right out of the box. By managing security compliance and infrastructure scaling, the platform helps you move your AI projects from pilot testing to enterprise-wide deployment while avoiding performance bottlenecks.

The intelligence tier: Red Hat OpenShift AI

Resting directly upon that foundational infrastructure is Red Hat OpenShift AI. This is the primary operational hub where data scientists execute their day-to-day tasks. Instead of forcing teams to toggle between disconnected applications, this comprehensive machine learning operations (MLOps) platform brings the essential tools for data processing, model training, and inferencing into a single unified environment. This acts as the vital link transforming raw code and experimentation into fully deployed, scalable AI solutions.

The unification hub: DagsHub

DagsHub gives AI teams a GitHub-like workspace for managing datasets, annotations, experiments, models, and code. It integrates familiar open source tools including Git, Data Version Control (DVC), MLflow, and Label Studio, so teams can track the full lifecycle of an AI project in one place, making collaboration and reproducibility significantly easier. DagsHub has built-in, granular role-based access controls (RBAC), and is accessible through a convenient web interface, a command-line interface (CLI), and a Python package built for machine learning developers.

End-to-end AI development platform with DagsHub and OpenShift AI

Integration of DagsHub with OpenShift AI creates a powerful and streamlined platform for AI development. OpenShift AI, a comprehensive MLOps platform, provides the infrastructure and tools necessary for deploying and managing AI/ML workloads at scale, including data processing, model training, and inferencing. By combining DagsHub's capabilities for version controlling data, models, and experiments with OpenShift AI's operational strengths, data scientists and MLOps teams can achieve end-to-end reproducibility, traceability, and efficient collaboration throughout the entire machine learning lifecycle. 

Figure 1: DagsHub with OpenShift AI architecture

Benefits of integrating DagsHub with OpenShift AI

• Streamlined deployment: OpenShift AI and DagsHub have partnered to deliver a comprehensive development environment specifically designed for data science workflows. This integrated platform empowers data scientists throughout the entire data science lifecycle.
• Dataset management and annotation: DagsHub provides dataset versioning, data exploration, annotation workflows, and collaboration capabilities, helping teams manage datasets directly within their OpenShift AI environment, supporting improved data quality and traceability.
• Minimized friction: Since we're able to develop end-to-end with OpenShift, we don't need to worry about the obstacles and delays that often arise when development and operations teams are not well integrated. This unified approach eliminates the need for separate tools and processes, simplifying workflows and increasing overall productivity.
• Strong customer adoption: Many organizations are already using the combined power of DagsHub and OpenShift AI to streamline their ML workflows and accelerate AI innovation.
• Shared credentials: Enables a unified authentication system across all components (such as OpenShift AI, DagsHub, MLflow, DVC, and Git) to simplify access and enhance security.

Get started today!

If you have access to an OpenShift cluster with OpenShift AI installed, head over to the AI quickstart and follow the instructions to quickly bring up DagsHub on the cluster and run through a demo. If you don't have an environment, reach out to your Red Hat contact.