Make every GPU-hour count: Progress tracking in Red Hat OpenShift AI

It's Friday evening. Priya, a machine language (ML) engineer at a financial services firm, queues up a fraud-detection fine-tuning job on an GPU cluster costing $55 an hour. The run should take about 40 hours—roughly $2,200 in compute. She double-checks the hyperparameters, submits the job, and heads home for the weekend.

Monday morning, she opens her laptop. The model had stopped learning sometime Friday night, but the job kept running—burning through 2 full days of GPU time on a training run that was going nowhere. That's over $1,500 in wasted compute, and she has to start over.

If this sounds familiar, you're not alone. Every practitioner who has run a multiday training job knows the uncomfortable questions:

• Is the model actually learning anything?
• Should I stop early and try different hyperparameters?
• Will it finish before the stakeholder review on Tuesday?

The cost of not knowing is real. On demand GPU clusters can cost upwards of $50 per hour. A single multiday training run can easily reach thousands of dollars, and a misconfigured run can waste an entire weekend of compute before anyone notices. In shared clusters, the impact multiplies: a stalled job doesn't just waste one team's budget—it blocks other teams from accessing GPUs they need. One team's blind run becomes another team's delay. The problem isn't that teams are careless. It's that they're flying blind. Every GPU-hour matters—and without real-time visibility into training progress, too many of them are wasted.

The observability gap

Here's the uncomfortable truth—from the platform's perspective, a training job that's converging beautifully and one that's completely stalled look identical. Both are pods in running state, consuming the same resources, showing the same green status indicator. Kubernetes has no concept of whether your model is learning.

Today, monitoring training progress is possible—but it's harder than it should be.

• Logs are scattered and hard to use: In a distributed training job, logs are spread across multiple pods. Finding the relevant output means knowing which pod to look at, how to access it, and how to interpret framework-specific log formats. That requires Kubernetes expertise, an extra skill set that most data scientists and AI engineers shouldn't need to acquire just to monitor their training runs. Bad and unstructured logs aren't easily machine-readable, so they can't drive automation.
• External tools solve observability but add management overhead: Experiment tracking platforms like MLflow and Weights & Biases are excellent for what they do. But they require additional infrastructure to deploy and maintain, API keys and access controls to manage, and they operate above the platform layer. They're also disconnected from the Kubernetes API, meaning it’s harder to integrate them with custom Kubernetes-native workflows and custom operators. They tell you what happened inside the training loop, but they can't tell the platform what's happening.
• Every framework does it differently: PyTorch DDP, FSDP, DeepSpeed, JAX—each has its own logging conventions, its own metrics format, its own way of reporting progress. There's no consistency across frameworks, which means there's no unified view for platform teams managing a heterogeneous training environment.
• There's no standard API: Until now, training jobs have had no standardized way to report progress back to the platform. The result is that platform administrators managing shared GPU clusters are left guessing which jobs are progressing, which are stalled, and which are about to finish. Without that visibility, capacity planning and troubleshooting are guesswork.

Closing the gap: Progress tracking in Red Hat OpenShift AI

Red Hat OpenShift AI now includes production-ready progress tracking for distributed training jobs, built into Kubeflow Trainer v2 and generally available beginning with Red Hat OpenShift AI 3.4. You can see what's happening while it's still happening—and act before it's too late. This feature directly addresses the observability gap by giving data scientists, ML engineers, and platform administrators real-time visibility into how training jobs are performing.

This visibility rests on 3 pillars:

1. Dashboard visualization: Real-time progress metrics are visible directly in the Red Hat OpenShift AI dashboard. At a glance, you can see progress percentage, current step and total steps, current epoch, estimated time remaining, training loss, and evaluation metrics for your TrainJobs. No separate tool, no additional setup.
2. SDK integration: The same progress data is available programmatically through the Kubeflow SDK. This enables teams to build automated pipelines that react to training progress—triggering early stopping, sending notifications, or reallocating resources based on real-time metrics.
3. Multi-framework consistency: Progress tracking works the same way whether you're using PyTorch DDP, FSDP, DeepSpeed, or JAX. It also supports custom training code through the CustomTrainer API. This provides a single unified experience across frameworks.

Seeing your training in motion

The best way to understand progress tracking is to walk through what using it actually feels like. There are 3 steps that matter.

1. Start your training job: You submit a training job from a notebook using the Kubeflow SDK, exactly as you would today. If you're using HuggingFace Transformers, progress tracking is automatic—no code changes needed. The integration detects the Kubeflow environment and begins reporting metrics from your training loop.
2. Watch its progress: As training runs, real-time metrics appear in the Red Hat OpenShift AI dashboard: progress percentage, estimated time remaining, completed steps, epochs, gradient norm, training loss, learning rate. No log tailing, no SSH-ing into pods, no external tools to set up.

Figure 1: Red Hat OpenShift AI dashboard shows a list of jobs with a progress bar indicating 40% completion for a running TrainJob.

Figure 2:  A detailed side panel for the job progress and real-time metrics.

3. Act on what you see: This is where progress tracking actually saves your time and compute. Notice the loss isn’t improving? Stop the job early and save hours of GPU time. See a run converging faster than expected? Free up resources for your teammates. Know exactly when a job will finish so you can plan around it—no more guessing whether the results will be ready for Tuesday's stakeholder review.

The key characteristics that make this practical:

• Zero setup: No additional infrastructure, no external dependencies, no extra configuration. Progress tracking works out of the box.
• Minimal overhead: Metrics are reported at natural training checkpoints without impacting training performance.
• Works across frameworks: Whether you're using PyTorch DDP, FSDP, DeepSpeed, JAX, or custom training code, the experience is the same.

Who benefits

Data scientists and ML / AI engineers see real-time training loss, progress percentage, and ETA without SSH-ing into pods or tailing logs. Detecting divergence or stalled training early—catching a stalled job on Saturday morning instead of Monday saves hours of wasted GPU time. Compare training runs at a glance from the dashboard to quickly identify which hyperparameter configuration is performing best.

Platform administrators get a single dashboard view across all training jobs in the cluster. They'll see consistent metrics regardless of whether a job uses DDP, FSDP, or DeepSpeed, and make better capacity planning decisions by understanding which jobs are progressing, which are stalled, and which are near completion.

In multi-tenant GPU environments where multiple teams share the same cluster, this visibility directly drives better resource utilization and ROI. For organizations building a GPU-as-a-Service model, progress tracking fills a critical gap—turning the "GPU utilization black box" into an observable, manageable system. Ultimately, this empowers platform teams to remove infrastructure bottlenecks and confidently hand data scientists the resources they need to accelerate AI innovation.

Beyond monitoring: Building a foundation for smarter training

Progress tracking is more than a monitoring feature—it's a foundation. Once the platform has access to real-time training progress data, a cascade of smarter behaviors becomes possible. While these advanced capabilities are not yet Red Hat product commitments, the foundation is in place. The upstream Kubeflow Trainer community is actively exploring several directions:

• Smarter hyperparameter tuning: Today, hyperparameter optimization engines like Katib rely on parsing logs with fragile regex patterns to evaluate trial performance. With training metrics available directly in the Kubernetes API, Katib can read training status natively—enabling tighter integration and more efficient trial orchestration. This is the KEP in discussion in the upstream community.
• Elastic scaling: Convergence speed and ETA data could inform scale-up and scale-down decisions. A model converging quickly could release GPUs for other teams, while one stuck in a plateau could request additional workers. The upstream community is exploring elastic PyTorch support in TrainJob to make this possible.
• CLI progress visibility: kubectl get trainjob with a built-in PROGRESS % column would bring training progress directly to the command line, giving operators instant visibility without needing the dashboard or SDK.
• Smarter checkpointing: While OpenShift AI already supports periodic and just-in-time model checkpointing, having ETA and progress data available, the platform could make smarter decisions about when to checkpoint—for example, saving state automatically based on job ETA. The upstream community is exploring transparent GPU checkpointing with CRIU to enhance TrainJob fault-tolerance and resiliency

These are areas of active exploration in the Kubeflow project—not product commitments, but they illustrate why progress tracking matters beyond the immediate feature: it creates the data layer that the entire training stack can build on.

Progress tracking for distributed training is generally available in Red Hat OpenShift AI 3.4. For more information, read the Red Hat OpenShift AI documentation. Or experience it yourself by starting a 60-day trial of Red Hat OpenShift AI.