A few weeks ago I was watching a coding agent run through a multistep task, and what stood out was not whether it got the answer right. I found, 9 times out of 10, it did. What stood out to me was the waiting. Every step meant another call to the model, another few thousand tokens of reasoning, another pause before the agent could act on what it learned. The entire loop was painfully slow.
That gap is the story of this moment. The newest wave of models, including NVIDIA Nemotron 3.5 Lightning, Google’s Gemma 4, Alibaba's Qwen3.8 Max, DeepSeek V4, Moonshot's Kimi K3, and the latest flagship models from OpenAI, Google, and Anthropic, are all good at the benchmarks used to separate them. When most models can do the work, "which one is smartest?" stops being the interesting question. For anyone serious about using agents in production, a better question comes to light: "Where does the balance between capability and speed sit, and how do you decide for a given task?"
The issue with benchmarks
For years, model choice was mostly a leaderboard exercise. Pick the top score and move on. That approach is running out of resolution. Legacy tests like Grade School Math 8K (GSM8K), Massive Multitask Language Understanding (MMLU), and HumanEval now cluster near the ceiling for frontier models, and researchers have started writing about benchmark saturation as a measurement problem: When the gaps between models shrink below the noise in the test, the ranking tells you less than it appears to.
Artificial Analysis, which tracks models across a combined intelligence index, has responded by plotting intelligence against multiple axes. It publishes intelligence versus output speed, intelligence versus time per task, and intelligence versus cost per task. That framing matters. It treats raw capability as 1 property among several, not as the whole scorecard. The question shifts from, "How smart is this model?" to "Where does it sit on the curve, and is that the right spot for my workload?"
Agents turn a small latency cost into a big one
Speed matters more for agents than for chat, and the reason is structure. A chat interface wraps a single model call, whereas an agent encapsulates multiple. A moderately complex task using a reason-and-act loop can make tens of calls, each depending on the result of the last, because the agent can't take its next step until the current tool or model response comes back.
The numbers compound more quickly than most of us imagine. Consider a 20-call task. On a slow backend, roughly 2 seconds per call adds up to something like 45 seconds of wall-clock time. On a fast backend, well under 1 second per call brings the same task in around 13 seconds.
This is why fast iteration has become a design goal, not a nice-to-have. A developer sitting at the keyboard waiting on a coding agent has an attention threshold. Cross it, and the tool stops feeling like a collaborator. Andrej Karpathy made a related point in 2025: As models get more capable, the human's job shifts toward orchestrating and reviewing the work of agents, and when your agents are slow or idle, you become the bottleneck.
The tradeoff
There's a genuine cost to intelligence, and it's quantifiable. Epoch AI studied the accuracy and runtime of models across major providers and found cutting a model's error rate in half tends to slow it down by roughly 2 to 6 times, depending on the task. On Graduate-Level Google-Proof Q&A (GPQA) Diamond, the factor was about 6; on a competition math benchmark, it was closer to 2. The models on the efficiency frontier were overwhelmingly the smaller, distilled variants, the ones with "flash," "mini," or "nano" in their names.
In the graph below, you can see the same shape in today's models. Kimi K3 sits near the top of the intelligence index but generates around 40 tokens per second, which Artificial Analysis flags as slow. Qwen3.8 Max is similar: High intelligence, roughly 47 tokens per second. At the other end, Nemotron 3.5 Lightning trades some intelligence for speed and is described as built for a different point on the frontier—a fast execution layer rather than a peak-reasoning model. And a few models manage to be both quick and capable for their tier: Google's Gemini 3.7 Flash and StepFun's Step 3.7 Flash both land on the speed-versus-intelligence frontier, fast enough for tight loops while still scoring relatively well.
The takeaway isn't solely that fast models are better, but that speed and intelligence are now separate dials, and the user should be aware of what model shines in which area for a truly optimized agentic workflow.
When the smarter model still wins
It'd be easy to read all this as "pick the fast model." I feel like that'd be unfair to say, as there are clearly benefits on both sides of the table.
Errors compound across a chain the same way latency does. A model that's right 90% of the time on a single call can look far less reliable when you measure whether it succeeds across many repeated runs of the same task. Benchmarks that test this, like Sierra's tau-bench, show consistency dropping as the number of attempts rises.
So the honest version of the tradeoff is this: For a self-contained, well-scoped step where a mistake's cheap to catch, speed usually wins. For a high-stakes decision early in a long chain, where an error cascades, the more capable model can be the economical choice even though it's slower. And often the best fix for an accuracy-critical step isn't a bigger model at all, but better architecture around it: Schema validation before a tool runs, checkpoints, and a reviewer step that catches bad output before it spreads.
So where do you draw the line?
There's no single cutoff, but there's a usable rule. Match the model to the role, not to the leaderboard. Route the fast, capable model to the high-frequency, low-stakes steps that make up most of an agent's work: Reading, routing, drafting, and simple tool calls. Reserve the slower, more capable model for the few steps where getting it right the first time protects everything downstream. Then measure the result on your own traffic, because a leaderboard median isn't your workload.
You don't always have to choose
The framing of speed against accuracy assumes the model comes as it is. It doesn't have to. A capable model can often be made faster on your own infrastructure without giving up quality.
Speculative decoding is the clearest example. A small draft model proposes several tokens ahead and the full model verifies them in 1 pass, and the accepted tokens are guaranteed to match what the full model would have produced on its own. It's a speedup, not an approximation. In vLLM, using the open source Speculators library and EAGLE-3-style drafts, teams have reported latency reductions in the range of 1.5 to 3 times, with the largest gains at lower request rates.
Quantization is the other lever. Tools like LLM Compressor compress a model, which cuts memory and latency, and published examples show 8-bit floating point (FP8) models recovering more than 99% of the original accuracy on common evaluations. Serve the result on vLLM, and benchmark it against your real request patterns with GuideLLM, measuring time to first token, inter-token latency, and throughput under load profiles that look like your production traffic rather than a synthetic ideal.
That's the point we keep coming back to. The interesting question is no longer which model is smartest. It's which model, tuned and served the way you serve it, is smart enough and fast enough for the job in front of it. The models are ready. The decision is now an engineering one.
Want to make a capable model fast on your own infrastructure? Explore the open source inference tooling behind Red Hat AI, including vLLM, Speculators, LLM Compressor, and GuideLLM, or see how they come together in Red Hat AI Inference.
关于作者
Sawyer Bowerman is an AI Developer Advocate on Red Hat’s AI team based in Boston, MA. He specializes in high-performance model serving and inference, focusing on scaling open source ecosystems like vLLM and llm-d to make large language models more efficient and accessible for developers. He is dedicated to bridging the gap between raw model performance and real-world developer productivity through open-source innovation.
