The last mile problem in agentic AI: Why tool calling reliability is harder than it looks

Most of the attention in agentic AI goes to the agent's reasoning capabilities. The model can figure out the right plan, break a goal into steps, and decide what to do next. Far less attention is given to the last step of that process, actually executing the action the model decided on. This “last mile” is taken for granted despite being the step that frequently causes agentic systems to quietly break in production.

What tool calling actually unlocks

Tool calls give your agent hands. It's the difference between a model that can only describe what it would do, and one that actually goes and does it, whether querying a database, filing a ticket, running a calculation, calling another service, and coming back with a real result instead of a guess. For an agent, tool calling isn't one feature among many, it's the entire mechanism through which reasoning becomes action. 

That also makes it prone to failure. Each step's output feeds the next step's input. So an agent that plans 5 steps and can only reliably execute 3 of them is as useful as one who doesn’t have access to any tools at all. 
 

The formatting problem 

Every model calls tools in a slightly different way. If the tool call isn't formatted exactly right, the entire response can fall apart. 
 

# Model A: XML-style tag wrapping a JSON payload
<tool_call>{"name": "get_weather", "arguments": {"city": "Boston"}}</tool_call>

# Model B: plain JSON, different field name for the same concept
{"tool_calls": [{"function": {"name": "get_weather", "parameters": {"city": "Boston"}}}]}

# Model C: a special token followed by a function-call-style string
[TOOL_CALL] get_weather(city="Boston")

Fig. 1: XML-style tags wrapping JSON, plain JSON, and function style tool call.

Some models use specific tags or tokens to mark the call, others require a particular JSON shape and field names. A parser has to know, ahead of time, which shape a given model is going to produce. This is made more difficult by models that mix chain-of-thought text in with the actual tool call, so the part of the response needed for the call is sitting right next to reasoning text that isn't meant to be parsed as instructions.

Why this hits agentic systems harder than chat

In a single chat exchange, a badly parsed tool call is one bad answer. In an agent working through a multistep task, it's the first domino. The next step's plan depends on what the previous step actually did. It doesn’t matter what the model intended to do, only how the tool call was actually parsed and executed. Get one step wrong along the way, and everything downstream reasons based on a world that doesn't match reality, with no obvious signal that anything went wrong.

That compounding is the real last-mile problem. All of these common failures break quietly on their own:

• Incorrect tag style: A parser tuned for one model's tag style will simply miss a tool call from a model that doesn't use tags at all. The call never fires, and the agent's plan silently loses a step.
• Mismatched labels: A parser expecting the label "arguments" won't necessarily choke on a payload that uses the label "parameters" instead. It can just quietly pass along an empty or wrong set of values, so the tool runs, just with the wrong input, and the agent proceeds as if that step succeeded.
• Varying call loads: A parser built to expect one tool call per response can grab the first call in a multicall response and silently drop the rest, so an agent that asked for 3 things only gets 1 back.
• Reasoning mixed with the call: When a model interweaves reasoning with the tool call and the parser isn't drawing that line correctly, it can pull arguments out of the reasoning text instead of the actual call. The next step of the agent's plan will be fed with values it didn't intend to send.

An agent doesn't have a human reading over its shoulder at every step, checking that each tool call did the right thing before moving on. That's the point of an agent. It's also exactly why a silent failure at the last mile is so much more expensive here than anywhere else in the system. Nothing stops a failure from dominoing into chaos, carefully constructing a multistep plan that's now confidently wrong.

Why this keeps breaking silently, even after it worked

Swap in a different model for cost or performance reasons, and an agent that was working perfectly can start silently failing to call tools at all, or start calling them with the wrong data. Most likely, nothing in the logs looks like an obvious error, because nothing crashed. It just quietly executed the wrong action, or no action, instead of behaving according to plan.

An update to a model you're already using can do the same thing—a minor change to how that model formats its own tool calls, and a parser that was hardcoded to the old shape breaks the moment the update rolls out. The failure may materialize as if it came from somewhere else entirely, buried under 3 or 4 agent steps that all ran on top of it.

What serving needs to do about it

Every serving engine needs to know how to read the tool call format each model produces and be intelligent enough to not break if the model version changes. Your serving engine should be able to:

• Recognize and correctly parse each model family's native tool call format, including tag-based, JSON-based, and special-token variants
• Normalize field name differences (arguments versus parameters, and similar) instead of assuming one shape
• Handle multiple tool calls in a single response, not just the first
• Separate reasoning text from the actual tool call before extracting arguments
• Stay correct across model version changes, so an upstream model update doesn't silently break a pipeline that depends on it

These are all needed for agentic systems. It's the difference between an agent that keeps working as models change, and one that silently stops doing what it's supposed to do the next time somebody upgrades a dependency.

How is your team handling tool call reliability in agentic pipelines today? We'd like to hear about it. Take a look at our what’s new and what’s next across Red Hat AI!