For an enterprise, AI sovereignty starts with this primary requirement: Retain control of its own data and infrastructure. But as organizations move from experimenting with AI to deploying it in real business processes, control over the models themselves has become increasingly important. This includes the ability to fine-tune and adapt models to the specific work the enterprise needs them to do.
Atos is doing exactly that with Red Hat AI, in a project focused on parsing German-language social media content across 14 categories, such as indicators of harmful content. To remain reliable for downstream systems, the model must deliver its insights in a consistent, defined JSON structure. At the same time, the solution must meet strict sovereignty requirements, with control over the data, models and training pipeline.
For this to work, Atos had to find an approach that could deliver the required accuracy and reliability, while also remaining both sovereign and economically viable at scale.
When more context isn't the answer
RAG has become an important way for enterprises to connect AI models to proprietary information. But not every problem can be solved by giving a model more context.
For Atos, the challenge was getting a model to reliably perform a specific task: Interpret primarily German-language content, extract the right information, and consistently return it in the required structure. That requires changing how the model behaves rather than simply what information it has access to.
With Red Hat AI, Atos was able to use synthetic data generation to create the high-quality datasets needed to fine-tune small models for structured extraction.
This addresses the scarcity of domain-specific data (a common fine-tuning hurdle), allowing enterprises to generate synthetic training sets needed to specialize a model for a particular enterprise task.
This gives an enterprise another way to connect its data to AI. RAG can provide models with relevant enterprise knowledge, and fine-tuning can be used when the goal is to teach a model more specialized and consistent behavior.
Small models can change the economics of enterprise AI
Why do this rather than simply using a frontier model? At enterprise scale, model size matters. The Atos use case was designed with potentially millions of records per day in mind, making inference cost and latency important considerations.
The key challenge was to find how small a model could be and still perform the required tasks. Fine-tuning improved Phi-4 Mini's overall evaluation score by as much as 26.4% compared with its base-model baselines. The tuned model also learned to consistently produce valid JSON, reaching approximately 99% validity through fine-tuning alone, without relying on an additional harness to enforce the output structure. This is particularly important because it shows that the required output behavior was learned by the model itself.
More importantly, the smaller models were able to compete with GPT-4.1 on this specific task. The fine-tuned Qwen 2.5 7B model outperformed GPT-4.1 in the evaluation, while the fine-tuned Phi-4 Mini came very close to GPT-4.1 despite its much smaller size. Phi-4 Mini also outperformed NuExtract 2.0 8B, a model purpose-built for structured extraction and more than twice its size, by up to 22.6% in overall score.
This doesn't mean smaller models will outperform frontier models for every workload, rather it proves that the largest model isn't necessarily the right choice for every task.
For well-defined enterprise workloads, fine-tuning a small model delivers the right amount of precision without the overhead of a general-purpose frontier model. For high-volume production, this dramatically improves the economics of running AI at scale.
From data sovereignty to AI sovereignty
Sovereignty isn't only about keeping data within a specific geographical border. Enterprises increasingly need control over the models themselves: Where they run, how they are adapted, and how they integrate into business processes.
Open models provide an important part of that control. An open model gives you the ability to customize models for your own workloads, and to deploy it on infrastructure you control.
This becomes increasingly relevant with agentic AI. Enterprise agents often orchestrate many specialized tasks, such as extracting information, classifying requests, producing structured outputs and interacting with enterprise systems. These tasks don't always require the broad capabilities of a frontier model. They often require consistent and task-specific behavior.
Rather than sending every task to a massive frontier model, you can choose the right model for each workload and specialize smaller models where it makes sense.
For Atos, the results prove that a small, tuned model can match or exceed frontier model performance on specific enterprise tasks, while delivering complete control required for sovereign AI.
Sovereign AI is not simply about where AI runs. It is also about giving enterprises the control and flexibility to shape AI around their own requirements.
What next
Want to explore what smaller, specialized models could look like for your AI workloads?
- Talk to a Red Hatter: Explore Red Hat AI or Contact sales directly
- Explore the platform: Explore Red Hat OpenShift AI
- Try hands-on testing: Try our AI interactive demos
- Read our technical blog post: Smaller model, sharper insights: How Atos approached structured outputs
- Get technical quickstarts: View AI quickstarts
About the authors
My entrepreneurial spirit led me to co-found an AI startup. This experience, combined with my work driving key go-to-market initiatives at Red Hat and building strategic partnerships, has shaped my ability to translate complex technologies into effective market strategies. I enjoy sharing these insights, whether speaking at UC Berkeley and Stanford or engaging with C-level executives. My background in AI research, including a collaboration between the Royal Institute of Technology and Stanford (with findings presented at SSDL 2017), continues to inform my passion for innovation.
Max Murakami startete seine Karriere in der akademischen Forschung am Frankfurt Institute for Advanced Studies (FIAS), wo er sich intensiv mit Computational Neuroscience und Spracherwerb beschäftigte. Der Wechsel in die Industrie führte ihn zur CID GmbH, wo er als ML Engineer die Brücke zwischen der Entwicklung und dem Betrieb von KI-Applikationen schlug. Seit seinem Wechsel zu Red Hat fokussiert er sich als AI Specialist Solution Architect darauf, Unternehmen beim Aufbau skalierbarer KI-Infrastrukturen und der Operationalisierung von Machine Learning (MLOps) zu unterstützen.
More like this
Announcing the general availability of MCP server for Red Hat Enterprise Linux
How Red Hat OpenShift 5 simplifies incident diagnosis with supervised AI
Can we trust AI agents in the enterprise?
Can agentic AI patch the internet?
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Virtualization
The future of enterprise virtualization for your workloads on-premise or across clouds