Stop paying for the same prompt: Optimize AI costs with Redis on Red Hat OpenShift

Large language model (LLM) API costs have a way of sneaking up on a business. What begins as a promising chatbot prototype often transforms into an invoice nightmare as users consume tokens at an unpredictable rate.

A significant portion of these costs stems from repetitive queries. Users often ask the same questions. "What is the return policy?" or "Can I send this back?" phrased in slightly different ways. Another hidden cost can be from questions that are not relevant to the application, topic, or business, or that may even be abusive.

Traditional caching fails in this situation because it requires an exact string match. If a single comma or word changes, the system treats it as a fresh, billable request to the LLM. To scale gen AI without breaking the bank, enterprises need a system that understands intent, not just characters.

Enterprise AI architecture with Red Hat OpenShift, Red Hat OpenShift AI, and Redis

Building a production-grade AI solution requires both a model and a coordinated AI environment where each component serves a distinct purpose.

Redis caching and routing architecture on OpenShift

Red Hat OpenShift: The reliable foundation

OpenShift provides the enterprise-grade Kubernetes orchestration needed to run AI workloads across hybrid cloud environments. Its role in is to make sure that the entire application, from the front-end UI to the backend databases, is scalable, protected, and resilient. It abstracts away the complexity of the underlying infrastructure, allowing developers to focus on the AI logic rather than the plumbing.

Red Hat OpenShift AI: The intelligent engine

OpenShift AI is the specialized platform for the AI model lifecycle. In a semantic caching architecture, its primary role is managing the embedding models and the LLM integration. It provides the environment to serve these models with optimized performance (using tools like vLLM), so the meaning extraction phase of your cache is as fast as the retrieval phase. 

This also works with models served from Models-as-a-Service (MaaS), whether on the internet or hosted locally using the MaaS functionality of OpenShift AI.

Redis: The semantic memory and intelligent routing layer

Redis plays a dual role in this environment, acting both as a semantic cache and as an intelligent routing layer. The cache handles near-duplicate queries by matching on intent, while a Redis-powered router sits alongside to classify and direct requests before they ever reach the LLM.

Repetitive queries can be served from cache, while out-of-scope or abusive requests, like users trying to turn a support bot into a coding assistant, can be blocked or redirected entirely. This routing is driven by offline mapping techniques that enable fast, low-cost decisions at runtime. 

While both components are powered by Redis, they serve distinct purposes: the cache optimizes response reuse, and the router enforces control. Together, they act as a guardrail system that reduces unnecessary LLM calls and helps keep costs more predictable.

The benefits: Why this approach wins

By combining semantic caching with intelligent routing, this architecture delivers both cost control and performance at scale:

• Cost savings: Avoid unnecessary LLM calls by serving repeat queries from cache, routing simple requests to cheaper models, and blocking irrelevant ones, significantly reducing total token spend.
• Improved speed: Cached and pre-routed responses return in milliseconds, dramatically faster than full LLM generation (often up to 4X improvement).
• Smarter workload distribution: The semantic router classifies requests up front, sending basic queries to lightweight models, complex ones to the full agent, and off-topic requests to exit—so the right resources are used for the right tasks.
• Higher cache hit rates: Semantic matching recognizes intent, not exact phrasing, so variations like “I forgot my login” and “How do I reset my password?” resolve to the same cached answer.
• More efficient infrastructure: Offloading repeat and low-value queries reduces load on expensive reasoning models, freeing capacity for novel, higher-impact tasks.
• Built-in guardrails: The same Redis-powered layer can block or redirect out-of-scope and abusive requests before they hit the LLM, enforcing boundaries and preventing wasted spend.

The power of the unified stack

While these tools are powerful individually, they work well together for 3 specific reasons:

1. Reduced latency: By hosting the Redis cache and the embedding models on the same OpenShift cluster, you minimize the "network hop" latency. This means the similarity search adds only  negligible overhead (5-20ms) while saving seconds of LLM processing.
2. Simplified architecture: With the Redis operator, you don’t need to stitch together separate vector databases and storage systems. It handles vectors, embeddings, and cached responses in a single product, with built-in automation, high availability, and lifecycle control.
3. Security and sovereignty: Running the entire environment on OpenShift allows enterprises to keep sensitive query data within their own security perimeter. You can cache and retrieve answers without sending repetitive, potentially sensitive data to external LLM providers.

Get started with our AI quickstart

Ready to stop paying for the same answer twice? We’ve made it easy to test this architecture in your own environment.

Try it out today. This AI quickstart will help you deploy Redis, OpenShift AI, and a sample application on OpenShift so you can see up the token savings for yourself.