AI agent observability: Building a production-grade operational layer

If you've been following along with our series so far, your AI inference layer is reliable. Your agents have cryptographic identity, scoped tool access, and guardrails at every boundary. The platform is production-grade. One question remains: how do you know any of it is working?

I keep coming back to the 6 AM incident that opened this series. Three failures hit a single AI agent deployment overnight—43 duplicate tickets, $4,000 charged to the wrong account, and a hallucinated refund policy that led to a $280 return the company had to honor. The agent ran on LangChain. It worked perfectly in development. Every failure was detectable. No one was watching any of them.

Bring your own agent (BYOA)—is Red Hat AI's approach: the platform provides production infrastructure for any agent framework without code changes. The gap between an agent working as intended in development and a production-ready deployment isn't a framework problem—it's an infrastructure problem. Earlier articles covered identity, connectivity, guardrails, and self-hosted inference. This article covers what happens after deployment—the operational layer that tells you whether your agents are doing what you think they're doing. Your framework handles the agent loop. It doesn't ship observability, evaluation, or cost governance. Without those capabilities, you're running autonomous systems blind.

When you can't see the reasoning

Traditional software is deterministic. Call a function with the same inputs, and you get the same outputs. When something breaks, you read the logs, find the error, and trace backward to the cause. Debugging is forensics with evidence.

Agents break that model entirely. A large language model (LLM)—the reasoning engine inside an agent—is stochastic. The same prompt can produce different tool calls, different reasoning chains, and different outputs on consecutive runs. When an agent fails, the output alone doesn't tell you why. You need the full chain: what the model received, what it considered, which tools it called, what those tools returned, and how the model interpreted the results. Without that chain, debugging isn't forensics. It's archaeology.

I've watched teams try to diagnose agent failures by staring at the final output and reasoning backward. It works about as well as diagnosing a car engine by listening to the exhaust.

Now consider the 6 AM incident through this lens. The 43 duplicate tickets happened because an application programming interface (API) call succeeded, but the response timed out—so the agent retried, and retried, and retried. With execution traces, an engineer would've seen the retry loop as a distinct pattern within minutes of it starting. The $4,000 wrong-account charge would have appeared in an audit trail tied to a specific agent identity—and the spend anomaly would have been visible before anyone noticed it on an invoice the following Monday. The hallucinated refund policy—90 days stated, 30 days actual—would have been visible as an output no evaluation layer had checked against documented policy. A customer returned a $280 product on day 47 citing the agent's response, and the team had to honor it.

For decision makers in regulated industries, this gap isn't theoretical. Financial services regulators and healthcare auditors increasingly require audit trails for automated decisions. "The AI did it" isn't an acceptable explanation when regulators ask how an automated system reached a conclusion. You need evidence showing what the agent considered, which tools it called, and what it produced. Observability isn't a monitoring upgrade—it's the evidence layer making your agent deployments defensible.

Tracing every step of the reasoning chain

That evidence has to come from somewhere. On Red Hat AI, it comes from MLflow tracing (an agent execution tracing tool, currently in developer preview, capturing every step of an agent's reasoning in a structured format you can search and replay).

What does "every step" mean in practice? MLflow records every turn of a multi-turn agent interaction: the user prompt, the model's internal reasoning, which tools the model decided to call, the exact API requests those tools made, the responses they returned, and how long each step took. Traces are structured and searchable. An engineer debugging the duplicate ticket failure would open the trace and see the 1st API call succeed, then see the retry trigger fire despite the success, then see the loop repeat 42 more times. The root cause—a timeout on a successful response—would be visible in the trace data, not buried in speculation.

I find this particularly valuable for the kind of failure showing up only at scale. A single retry looks like normal error handling. 43 retries look like a pattern that should have triggered an alert after the 3rd.

MLflow traces speak OpenTelemetry (an open standard for telemetry data—traces, metrics, and logs—that any observability platform can consume). This matters because agent traces aren't locked in a proprietary format. Any OpenTelemetry-compatible platform—Prometheus, Grafana, Jaeger, Datadog—can ingest agent traces alongside your existing service metrics. Your agents show up in the same dashboards as your APIs and databases. For developers, that means no new tooling to learn. For operations teams, it means agent behavior is visible in the monitoring stack they already trust.

Every trace captures token usage and tool execution costs, tied to the specific agent, workflow, and team that generated them. When an agent starts generating unexpected API charges, the cost spike appears in the trace data before the invoice arrives. That $4,000 wrong-account charge from the 6 AM incident? Cost tracking would have surfaced the anomalous spend pattern as it happened—not the following Monday when someone reviewed a billing statement.

What to ask about your own agent deployment

Before moving further, it's worth pausing on a set of questions I think every team running agents should be able to answer:

• If 1 of your agents starts retrying a successful API call, how long before someone notices?
• Can you trace a specific customer interaction back through every tool call and model decision that produced it?
• When a model update changes your agent's behavior, do you discover it from your evaluation pipeline or from a customer complaint?
• Do you know which agent, workflow, or team is responsible for a given spike in inference costs?

If the answer to more than 1 of these is "I'm not sure," the operational layer is the gap. These aren't hypothetical concerns—they map directly to the failure modes the 6 AM incident demonstrated.

Staying correct after deployment: Continuous evaluation

Tracing tells you what happened. Evaluation tells you whether what happened was right—and that's the harder problem.

Static test suites catch regressions at deployment time. Run your tests, verify the outputs, and deploy with confidence. The trouble is agent behavior drifts. A model update, a change in a downstream tool's response format, or a shift in the data the agent encounters can cause a previously correct agent to start producing incorrect outputs. The test suite passes. The agent fails. Nobody notices until a customer does.

This is the gap continuous evaluation closes. EvalHub (Red Hat's open source evaluation control plane for agent quality and safety—an orchestration layer coordinating adversarial scanning, guardrail evaluation, and quality scoring through a single API call on OpenShift AI) moves beyond static test sets into online evaluation, scoring live production traffic against quality and correctness criteria in real time.

The technique at the center of this approach is LLM-as-Judge (a method where a capable LLM scores another model's outputs for quality, faithfulness, or safety—enabling continuous evaluation without requiring human review of every response). A judge model evaluates what the production agent produces against known-good criteria, policy documents, and factual references. When an output scores below threshold, the system can flag it for human review before it reaches the end user or downstream system.

Think about how this would have caught the hallucinated refund policy. The agent told a customer the refund window was 90 days. The actual policy was 30 days. The output was plausible, grammatically correct, and confidently stated—a human skimming it might not have caught it either. A judge model with access to the actual refund policy document would've scored it low on faithfulness and triggered a review step. The customer never sees the wrong answer. The $280 return never happens. The legal exposure from an agent making unauthorized contractual representations never materializes.

I think of continuous evaluation as the difference between asserting your agents produce correct outputs and demonstrating it. For compliance teams evaluating requirements like General Data Protection Regulation (GDPR), Health Insurance Portability and Accountability Act (HIPAA), System and Organization Controls 2 (SOC 2), or Federal Risk and Authorization Management Program (FedRAMP), that distinction is everything. Assertions are promises. Demonstrations are evidence.

Catching vulnerabilities before they reach production

Continuous evaluation monitors agents already deployed. There's an earlier checkpoint that matters just as much: testing how your agent behaves under adversarial conditions before it ever sees production traffic.

Well-formed inputs are the easy case. The hard case is deliberate attempts to jailbreak the agent, trick it via prompt injection, or extract sensitive information it shouldn't reveal. An agent performing well on your test suite may still be vulnerable to attacks your test suite never imagined.

NVIDIA Garak (automated adversarial vulnerability scanning for agents, currently in developer preview in Red Hat OpenShift AI 3.4) runs as part of the deployment pipeline and integrates with TrustyAI (the operator coordinating safety controls on Red Hat AI). Think of it as a penetration test for your agent—the same principle as security scanning in a continuous integration and continuous delivery (CI/CD) pipeline, applied to agent safety. Garak probes for jailbreaks, prompt injection vulnerabilities, and information extraction attacks, then reports what it found before the agent reaches production.

I see this as the natural extension of what development teams already do for application security. Nobody ships a web application without running a vulnerability scanner. Agents interact with tools, APIs, and customers with the same autonomy as a web application—and a broader attack surface. For teams operating in regulated industries—financial services under SOC 2, healthcare under HIPAA, organizations subject to the European Union (EU) AI Act—pre-production adversarial scanning is moving from best practice to requirement. Garak surfaces vulnerabilities before they become incidents. For developers, it integrates into the workflow they already know: run the scan, review the results, fix the findings, deploy. For empirical results from running Garak probes against a real agent across 3 hardening tiers, see Testing infrastructure red teaming with obliterated models on Red Hat Developer.

One area where the industry hasn't yet converged on a solution: agent memory and persistence poisoning. Attacks instructing an agent to write attacker-controlled content into its own memory or configuration files are structurally difficult to detect—updating memory is indistinguishable from a legitimate request. Open Web Application Security Project (OWASP) added Memory and Context Poisoning as ASI06 to the 2026 Top 10 for Agentic Applications, and research from NeurIPS and Microsoft confirms real-world exploitation. Sandbox isolation, network policies, and prompt injection classifiers all fail to address this vector. It's an active area of research, not a solved problem—and teams deploying agents with persistent memory should treat it as an open risk.

The beginning, not the end

The 6 AM failures weren't unforeseeable. They were unobserved. Every 1 of them left traces that would have been visible if anyone had been watching—retry loops piling up, spend anomalies spiking, and customer-facing outputs contradicting documented policy. The agents weren't broken. The operational infrastructure to watch them didn't exist.

Deploying an agent isn't the end of the engineering work. It's the beginning. The difference between organizations scaling agents and those quietly rolling them back is whether the infrastructure exists to detect problems before they reach customers. And when a regulator or auditor asks how an agent reached a specific decision, the evidence needs to exist on infrastructure the organization controls—not in a third party's logs. That's the operational dimension of sovereignty: not just owning the model and the data, but owning the audit trail. The platform capabilities are now in place—identity, connectivity, inference, observability, and evaluation. What remains is bringing them together. How does a team go from experiment to production?

Get started

Ready to build the operational layer for your agents?

• Try OpenShift AI free in the Developer Sandbox: Experiment with MLflow tracing and evaluation in a preconfigured environment
• Explore Red Hat AI interactive demos: Observability and evaluation walkthroughs
• Read Red Hat AI documentation: Agent execution traces, OpenTelemetry integration and cost tracking
• Explore Red Hat AI for developers: Trials, demos and links to MLflow and Garak upstream projects
• Take the free Red Hat AI Foundations course: Hands-on labs covering observability and evaluation
• Building trust through AI red teaming: Red Hat's approach to testing model safety: Red Hat's integrated safety stack: SDG Hub, Garak, NeMo Guardrails and Eval Hub in a single automated pipeline