As AI agents move from demos into production, teams are discovering that benchmark scores alone rarely translate into business value. Routinely, a new frontier model is released, climbs to the top of the benchmark leaderboards, and enters production, but business metrics and code velocity remain flat. We present 5 principles for evaluating agents beyond standard model benchmarks.

A new software era requires a new notion of correctness

Our definition of what makes software "correct" must change as we enter the era of software 3.0.

Software 1.0 (code is software): Unlocked the digital age. Correctness was straightforward because code explicitly defined every detail.

Software 2.0 (models are software): Advanced computer vision and chatbots. Correctness shifted to statistical metrics determined by data and model design.

Software 3.0 (prompts are software): Changed how humans interact with machines. However, we still lack a clear definition of correctness. Defining it is the key to unlocking true value.

Software 3.0 has 2 primary pain points:

Verification bottleneck: We have built machines that can generate solutions faster than we can evaluate them.⁷ The field's progress is now gated by our ability to construct verifiers. 

A recursive trust problem: Using large language models (LLMs) to evaluate LLMs creates a recursive trust problem.⁶ LLM-generated evaluators inherit all the problems of the LLMs they evaluate.

Although the first phase of AI was won by researchers, these pain points show that the next phase belongs to people who can define what "good" means for real-world tasks.⁹

The 5 pillars of AI evaluation systems

Agent evaluation systems can be broken down into 5 distinct areas.

1.  What are you actually measuring?

You think you're measuring the model, but you're actually measuring the stack, which contains 3 layered sources of variance.⁵

Differences in data and harness configuration can outweigh differences between models.

Harness dominance: In many modern benchmarks, the evaluation harness can introduce more ranking variance than changing the underlying model.² Brand shows command line interface (CLI) harnesses obliterate standardized state-of-the-art (SOTA)—same model, different harness, completely different score.²

Even when the harness is held constant, the data itself can undermine results. Ground truth is a claim, not a fact. It can be wrong, contaminated, or outdated. Building on incorrect ground truth produces precisely calibrated wrong conclusions. Together, harness and data effects compound into a third problem: elicitation. A model's score is a lower bound on its capability, never an upper bound.

2.  What should you evaluate beyond output?

What the evaluation examines. Lee identifies five; most teams cover only the first.⁴

Correct outputs can mask broken reasoning. Consider a coding agent tasked with fixing a failing test. It queries a documentation API, receives a 404, and instead of retrying or reporting the error, generates a plausible-looking patch from its training data. The test passes—but only because the fix papers over the symptom rather than addressing the root cause. Output evaluation says "pass." Trace evaluation reveals the agent never read the relevant documentation and hallucinated the fix. Without multisurface coverage, you will have huge blind spots.⁴

This problem extends beyond the agent's outputs into the systems it touches. Agents mutate their environment. Evaluation that captures only terminal state, not per-step deltas, cannot attribute failure to specific actions. The diff is the evidence; the final state is the verdict.⁴

3.  What infrastructure do reproducible evals require?

The infrastructure architecture of evaluation that is under your control.⁴

Agents don't need datasets—they need worlds.⁴ They need isolated filesystems and database snapshots. Experiments must never share mutable state. The data plane is the harder engineering challenge because it must provide reproducible, isolated, instrumentable environments.

Providing these worlds only matters if each trial starts clean. Without trial isolation, you measure infrastructure artifacts, not agent capability.¹ Anthropic caught Claude exploiting Git history from previous trials.¹ Shared state between runs produces correlated failures that corrupt results. Every test run needs a completely clean environment.

4.  Which metrics should you optimize for?

Depending on your goals, success can be defined by 4 distinct metrics. ^{1, 3, 7}

Verifier's Law: The ease of training AI to solve a task is proportional to how verifiable the task is. Real-world tasks lag because there is no obvious verifier for "useful." The field's progress is gated not by model capability but by our ability to construct verifiers.⁷ 

We see this pattern in software. New discrete features are rapidly generated because unit tests are trivially verifiable while overarching goals requiring fundamental architectural changes are left unresolved. The result: more PRs are opened, maintainers are overwhelmed with reviews, and the real objectives are not met. On the other hand, better designed software leads to a fly wheel effect. A code base with a consistent, principled design maximizes discrete features that are architecturally sound, and therefore it will get much more utility from agents than a code base littered with heuristics and bandages.

Verifiability tells you whether improvement is possible; cost tells you whether it is practical. Accuracy means little without considering cost.³ Many "SOTA" results drastically overfavor capability versus cost. Joint optimization of cost and accuracy can greatly reduce cost with minimal impact on accuracy.3

5.  How do you improve over time? 

Improvement is an iterative feedback loop, commonly referred to as the evaluation-driven development (EDD) cycle.⁸

1. Error analysis (observe): Review trace logs to see simple pass or fail results.

2. Hypothesis (fiagnose): Identify a clear, testable cause for a failure pattern.

3. Experiment (test): Change 1 variable at a time.

4. Measurement (verify): Measure the results.

5. Criteria evolution (adapt): Update your grading rules as you spot new issues, then return to step 1.⁶

Criteria drift: You cannot write the rubric before you grade.⁶ Criteria depend on observed outputs—you need criteria to grade, but grading helps define criteria. This is not a bug; it's the nature of evaluating open-ended systems. Evaluation systems must accommodate iterative rubric evolution, not assume fixed rubrics.

This loop only works if teams commit to running it consistently. The scientific method applied to AI products is the differentiator—the sustained commitment to sample, annotate, experiment, and monitor matters more than any evaluation tool. In practice, this discipline matures in stages: begin by writing evaluators for errors you discover and progress to eval-driven development as failure modes stabilize.

Closing the gap between benchmarks and production

We can now look at the 2 primary pain points of software 3.0 through the lens of the 5 pillars. The recursive trust problem is mitigated by Pillar 2 and Pillar 3 by making sure no single evaluator or shared state is trusted alone. Pillar 4 and Pillar 5 tackle the verification bottleneck by treating verifier construction as the core engineering challenge, not an afterthought. The gap between benchmark performance and real-world value is not a model problem, it's an evaluation problem. To bridge it, treat evaluation as a first-class engineering discipline. The companies that internalize this will ship more reliable and cost-efficient agentic systems; the rest are more likely to build impressive demos that fail in production.

Build production-ready agent systems at scale

Moving AI agents from prototype to production requires robust data plane isolation, multisurface evaluation, and reliable cluster-level governance. Discover how Red Hat OpenShift AI provides the enterprise infrastructure needed to test, evaluate, and scale trusted LLM applications across your hybrid cloud. To implement EDD natively in your pipeline, explore how EvalHub orchestrates multiframework LLM evaluation jobs across Red Hat OpenShift clusters.

References

^1.  Anthropic, "Demystifying Evals for AI Agents," https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

^2.   Brand, "Benches 2026," https://florianbrand.com/posts/benches-2026

^3.  Kapoor et al., "AI Agents That Matter," https://arxiv.org/abs/2407.01502

^4.  Lee, "Hidden Technical Debt: Agent Evaluation Infrastructure," https://leehanchung.github.io/blogs/2026/06/13/hidden-technical-debt-agent-evaluation-infra/

^5.  OpenAI, "Trustworthy Third-Party Evaluations," https://openai.com/index/trustworthy-third-party-evaluations-foundations/

^6.  Shankar et al., "Who Validates the Validators?," https://arxiv.org/abs/2404.12272

^7.  Wei, "Asymmetry of Verification and Verifier's Law," https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law

^8.  Yan, "An LLM-as-Judge Won't Save the Product," https://eugeneyan.com/writing/eval-process/

^9.  Yao, "The Second Half," https://ysymyth.github.io/The-Second-Half/