Pluggable by design: An agent mesh for software modernization that adopts the next model release

In our previous post, we walked through the agent mesh for software modernization architecture we built on Red Hat AI for modernizing legacy systems at the scale that mission environments actually require. The post covered the harness pattern, the workflows the harness orchestrates, and the KPI framework for measuring success. It also explained why we run coding agents on Devstral and non-coding agents on Ministral, inside a platform that operates in disconnected environments.

This post is about the next layer of that story: the pluggability.

In the regulated environments where agent mesh for software modernization belongs, the question of which model to run is never a one-time decision. Codebases run to millions of lines while defense and financial services environments are often disconnected. This means whatever runs the workload has to keep running without an outside API call. The model landscape keeps moving, and at some point one of those moves will involve swapping out the model powering your coding agent entirely. The architecture has to absorb that without rebuilding the iterations around it. That is what makes pluggability structural rather than aspirational.

In agent mesh for software modernization, the model is a component, the agent is a component, and the static tools around them are components. The model swaps in through the serving layer. Agents and the static tools they call sit behind clear interfaces, so a connected commercial coding agent (e.g. Claude Code or Codex) can run early when policy allows it, or it can be swapped for an on-premises alternative like OpenCode or a custom agent without rewriting the orchestration above. The iterations of the harness, the automated workflow tracking and merge policy, and the audit trail stay stable across those swaps. Teams can start on the components that fit today and absorb what comes next as upgrades rather than rebuilds.

This post walks through what that looks like in practice when we upgraded to Gemma 4 and the latest CrewAI in parallel. We cover where the swap was clean, where it required scoped engineering work, and why this property of the architecture matters more in regulated environments than it does anywhere else.

Pluggable by design

Figure 1. Models, agents, and static tools are components. The capabilities, the branching policy, and the audit trail are the architecture.

Today, agent mesh for software modernization is more of a paradigm than it is a  product. It defines a pattern for how agents, tools, models, and orchestration frameworks coordinate under a governed architecture. The pattern is what makes the components swappable. The harness we built is the implementation of that pattern, and it's built on rails that treat the model and the framework as components rather than as load-bearing dependencies.

Under the hood, the harness organizes work as 5 capabilities. The first, code archaeology, produces the migration plan and other named assets (for example, a software bill of materials (SBOM), a list of risky modules, or answers to ad hoc queries about the codebase). It uses SDGHub for metadata generation and GraphRAG for structural indexing.

The  4 other capabilities are characterization tests, planning, change generation, and review. Change generation is where the full harness runs. It is tool first by default, using Red Hat's migration toolkit for applications and OpenRewrite for static refactoring. If no deterministic tool exists for a question, Change generation falls back to an agentic harness that runs as many iterations as the work requires. Each iteration focuses on one configurable concern: code, testing, evaluation, continuous integration (CI), specifications, documentation, or whatever else the team needs.

Within a short period of time, models change. Frameworks change. Tool calling protocols change. The rails are what matter. If your architecture treats any of those as commitments, every change becomes a rebuild. If your architecture treats them as components behind a stable contract, change becomes an upgrade. In agent mesh terms:

The model is a component. Served through vLLM and Red Hat AI Inference, swappable across the validated catalog without touching the iterations above.

The agent is a component. An agent is a model plus a runtime that wraps it. The mesh treats agents as swappable. A connected commercial coding agent can run today and be replaced by an on-premises alternative before disconnection, or replaced by a custom agent or non-AI tool at any point, by introducing a new driver script and updating the YAML the harness reads. The orchestration above does not need to change.

The static tools are components. OpenRewrite and migration toolkit for applications, with our implementation today, with OWASP scan or other potential analyzers expected in additional engagements. Added or replaced without rewriting the iterations that call them.

Orchestration is worth calling out separately. The mesh is built on CrewAI today, and we have already navigated a version upgrade to keep pace with the framework's evolution. Swapping it out for something else entirely is a different conversation, and not one we've had yet. That layer carries real coupling either way. Decoupling orchestration is a direction the community is actively exploring (LlamaStack and OGX have both made attempts), and one we are watching closely.

Everything else stays stable across these upgrades: the harnessed iterations, the automated workflow tracking and merge policy, the named assets the iterations produce, and the audit trail end to end. That stability is what lets the system compound across releases instead of restarting at each one.

Pluggability under live conditions with Gemma 4 and CrewAI

Models and frameworks move on their own schedules, and neither coordinates with enterprise procurement cycles. An architecture that treats any one of those as a fixed dependency ages out fast. The Gemma 4 addition and CrewAI upgrade that we incorporated after their releases are instances of how the mesh absorbs that cadence rather than fighting it. Here's what we did and what stayed stable around it.

Gemma 4 ships with native function calling built into the model through dedicated special tokens. It's the first Gemma family to treat tool use as a first class capability rather than something prompted into the model after the fact. Coupled with Gemma 4's configurable thinking modes and native system prompt support, this gives the harness a model that behaves predictably when asked to invoke tools and reason about them.

The latest CrewAI consumes those structured tool calls through its large language model (LLM) abstraction rather than through text parsing. That distinction matters more than it sounds. Text parsed tool calls are brittle, fail in ways that look like model errors, and require defensive code in the orchestration layer. Structured tool calls flow through a typed interface that the framework, the harness, and the audit layer can all reason about.

As a result, we made 2 distinct kinds of changes inside the harness.

The Gemma 4 swap was clean. We configured Gemma 4 through vLLM with its native tool call parser and the model specific chat template, and the capabilities above the model layer did not need to change. Code archaeology, characterization tests, planning, change generation, review. None were aware the model had moved, they just saw a better behaved model underneath.

The CrewAI upgrade wasn't especially difficult, but it did require thoughtful consideration. Moving from 0.x to 1.x was a major version change, and because agent mesh is currently coupled to its orchestration framework, the upgrade touched the mesh code itself. But this wasn't a painful rewrite. It was a scoped refactor driven by a clear market need for native tool calling, which the prior CrewAI version didn't support.

This is where the agent mesh for software modernization pattern helped. The model layer remained pluggable, the harness patterns held stable, and the branching policy and audit trail carried forward. The framework layer required adjustment, but the architecture above it stayed intact. That is what pluggability means in practice: not zero change everywhere, but clear seams, bounded refactoring, and stable patterns where they matter most.

Red Hat’s impact on the last mile of AI delivery

There's a part of this story that's easy to miss. Getting Gemma 4 to behave correctly inside vLLM, with the right chat template, the right tool call parser,  and the right runtime configuration, was not automatic. We had to work through chat template inconsistencies, tokenizer edge cases, and the real integration issues that show up when a model moves from a benchmark into an actual serving framework. This is the last mile of AI delivery, and it has to be done for every model release.

As a concrete example, Ben Browning on our team identified that the Gemma 4 chat templates weren'thandling OpenCode's system message structure correctly. The templates expected a string and OpenCode was sending a list of text content parts. The model was seeing JSON-rendered prompts instead of the system instructions. Ben posted a diff to the public OpenCode issue thread, then shipped a fuller fix that also resolved an empty content edge case in multiturn tool calling. The official Gemma 4 chat templates were updated upstream with those fixes within 24 hours, and now the next Red Hat customer running Gemma 4 on vLLM doesn't have to find that bug. You can read more about this here.

That integration work is what Red Hat AI does for the open source ecosystem. The model providers, including the largest frontier labs, don't always have the systems integration depth to resolve these issues across every platform and infrastructure combination their customers run. They rely on partners who do. Red Hat AI Inference is a product of that integration work, continuously, across every major model release.

The pluggable architecture of agent mesh works because the substrate underneath it works. The agents don't have to know that someone fixed a chat template last week, they benefit from it anyway.

Why this matters in mission environments

In defense-adjacent and other regulated environments, the architecture decision behind every AI system is whether change can be absorbed or whether change forces a rebuild. Requirements include disconnected operations, Federal Information Processing Standards (FIPS) validated cryptography, and audit requirements that demand every component be traceable.

In that world, an AI architecture that treats the model or the framework as a hard dependency will be obsolete in no time. Six months after standup, a better model arrives. Twelve months in, the orchestration framework has had 3 releases and the model has had 2. If absorbing those changes means rerunning the security baseline and revalidating the deployment from scratch, the system stops improving. It freezes at the version that was current the day it was approved.

Agent mesh for software modernization is one of the only approaches built for real constraints and real consequences. The pluggability makes the system durable in environments where you can't rebuild every quarter and where the cost of falling behind is measured in mission readiness.

Closing thought

Most AI architectures are commitments dressed up as designs. The model is wired in. The agents are wired in. Every release becomes an event, and every event becomes a rebuild.

Agent mesh for software modernization treats the model and the agents as components, governed by a harness that absorbs change instead of breaking under it. Gemma 4 swapped in clean. The CrewAI upgrade required scoped refactor work in the mesh code itself, and that is the honest state of the orchestration framework layer today. Both landed in production. Neither forced a rebuild of the iterations, the branching policy, or the audit trail.

That is the part that matters. A new model release will always get attention. A framework upgrade will always create work. But the real question is whether the architecture can absorb those changes without forcing the entire system to be redesigned around them. In this case, the model changed, the framework changed, and the core modernization workflow kept moving. The architecture is what decides whether you survive the next release and yet still propel the mission along. That is what agent mesh is meant to protect.