Stop burning your AI budget: Optimize GPU usage and model deployment with workflow navigator

Uber burned through its entire 2026 AI tools budget by April. Microsoft faced a similar crisis, pulling Claude Code licenses because the tool worked too well and people used it too much. Even OpenAI's chief executive officer (CEO), Sam Altman, has called token costs "a huge issue" for the company and its customers. "Tokenmaxxing," the tendency to burn through tokens without a clear link to business outcomes, has gone from an internal joke to a boardroom problem.

Companies are responding by capping external AI budgets and pulling licenses. Fair enough. But when teams try to escape these escalating costs by deploying models on their own infrastructure, they trade metered billing for a new hidden expense: infrastructure waste. And nobody is capping this second cost problem because most organizations don't even know how to measure it.

From token burn to infrastructure waste

Here's what's easy to miss. Per-token costs have dropped as much as 80% over the past year. That sounds like good news, and it is, until you look at what happened to total spending. It went up. Agentic workflows, where AI loops through planning, tool calling, verifying, and correcting, burn through tokens at rates chat-based interactions never approached. According to Deloitte's 2026 TMT Predictions, inference workloads now account for roughly two thirds of all AI compute, up from about a third in 2023. Cheaper tokens didn't make AI cheaper. They made it easier to spend more.

Someone picks a model. Usually whatever topped a leaderboard last month, or whatever a colleague dropped in Slack. Teams deploy it on graphics processing units (GPUs). The configuration is a best guess because nobody has time to benchmark multiple different setups. The autoscaling is either too aggressive, too conservative, or non-existent. The quantization settings are whatever the last tutorial or blog post read used.

Then the GPUs run, usually not very efficiently. The model could be significantly larger than what the task needs, or the hardware may sit underused because the deployment was sized for peak traffic that never showed up.

This is infrastructure waste. It doesn't show up in a line item the way an API bill does, but it adds up every day those GPUs are deployed for the model, regardless of its utilization. It's a common pattern across organizations deploying self-hosted models.

Where self-hosted deployments go wrong

Two decisions cause most of the waste.

Picking the wrong model

There are thousands of open weight models out there. The default is to grab the biggest, most recognized one. But bigger isn't often better for your specific task. Early work on model routing, such as the RouteLLM framework, suggests directing tasks to appropriately sized models, using smaller models for extraction and classification, and reserving larger ones for complex reasoning can cut total spend while preserving most of the output quality. Most teams never run that comparison. It can take days of digging through leaderboards, benchmark papers, and hardware spec sheets, and the results change every month.

Deploying it by guessing

Once a model is picked, someone has to configure the serving runtime, GPU allocation, quantization level, and autoscaling parameters. Without performance data specific to their hardware and traffic patterns, even reference configurations become guesswork. A configuration set too high can waste compute, while a configuration that's too low can cause latency issues. Either way, the team is paying the price—whether in wasted budget or a poor user experience.

The people who know how to get both of these decisions right are rare. Most teams don't have someone who can look at a model, a hardware profile, and a traffic pattern and know the right answer.

Eliminating deployment guesswork with workflow navigator

We built the workflow navigator tool for exactly this situation. Workflow navigator is a natural language copilot in Red Hat OpenShift AI capturing user intent and converting it to actionable workflows.

You tell it what you're building. Something like, "I need a model for a retrieval-augmented generation app, 20 concurrent users, latency under 1.5 seconds." Workflow navigator looks at your model registry, checks what GPU hardware is on the cluster, pulls benchmark data for your use case, and comes back with ranked options.

On the model selection side, workflow navigator filters against your real hardware capacity and ranks by the benchmarks mattering for your task. Say your team needs a coding assistant. The instinct is to deploy a 70 billion parameter model because it topped a leaderboard. Workflow navigator's planner runs the numbers and shows a 32 billion parameter model on 2 GPUs meets your latency targets and scores similarly on code benchmarks, while the 70B needs 4 GPUs for the same job. That's half the hardware for comparable results. Workflow navigator shows you that comparison before you commit GPU hours to the wrong model.

On the deployment side, workflow navigator generates production-ready Kubernetes configurations for your chosen model. Its capacity planner estimates GPU memory requirements across different quantization levels and tensor parallelism configurations, then the GPU recommender compares performance across different hardware without running actual benchmarks.

A team deploying a chat model might discover an L40 with 8-bit floating point (FP8) quantization delivers latency within their service level objective (SLO) at a fraction of the cost of an A100 running the same model at 16-bit floating point (FP16). Workflow navigator produces the KServe InferenceService YAML, resource requests, autoscaling rules, and observability config, all sized against what the cluster has. You look at the tradeoffs and pick the config fitting what you care about, whether that's cost or performance.

Tackling infrastructure waste, not just API bills

The cost conversation in AI is almost entirely about the API side of the bill. That's a real problem, and Uber and Microsoft are right to address it. But nobody is talking about the infrastructure side with the same urgency. You have models running on real GPUs, burning real electricity, and deployed with configurations never tested against the workload. That waste is harder to see, so it doesn't get fixed.

When you pick a model based on reputation instead of fit, and deploy it with default configs instead of tested ones, the cost difference can be significant.

The workflow navigator tool isn't trying to solve the API cost problem. It's aimed squarely at the infrastructure cost problem. When you deploy a model on your own hardware or on the cloud on OpenShift AI, workflow navigator helps you deploy the right one, configured for how you're going to use it.

Try it

Workflow navigator is available now as a developer preview as part of OpenShift AI. It uses Model Context Protocol (MCP), so it works with Claude Code, Cursor, Gemini CLI, or whatever MCP-compatible client your team prefers.

Workflow navigator is built from 2 open source projects. The rhoai-mcp server provides 90+ tools for managing OpenShift AI environments programmatically, from project setup and workbench management to model serving and pipeline configuration. The llm-d planner handles the intelligence layer, the conversational requirements gathering, SLO-driven capacity planning, GPU memory estimation, and multicriteria deployment ranking turning "I need a model for X" into a running, optimized deployment.

If your team is spending too much time picking models and tuning deployments, try it out. We're building this in the open and would love to hear your feedback.

Or learn more about Red Hat OpenShift AI.