AI: Red Hat's vision for an open source future

The AI landscape is evolving at an electrifying pace. Just as with any technological leap, the question arises: what path will best shape its future? Red Hat believes the answer is clear:

The future of AI is open source.

This isn’t just a philosophical stance; it’s an approach focused on unlocking the true value of AI and making it something far more accessible, far more democratized and far more powerful.

We have always believed in the power of open source development in driving innovation. We’ve seen this play out in the rise of Linux, KVM, OpenStack, Kubernetes and many other projects that have helped shape the technical landscape of today. Through collaboration, transparency and community-driven innovation, open source development models accelerate the pace of discovery, encourage experimentation and democratize access to cutting-edge tools and technologies. This leads to faster progress, greater innovation and vibrant ecosystems.

AI is no different.

In AI, where trust, security, and explainability are paramount, everyone — not just those with the deepest pockets or more extensive resources — should be able to participate. Red Hat is dedicated to championing open source AI innovation, paving the way for the future of this technology to be built on a foundation of community-driven development, shared progress, and choice.

We’re investing heavily in open source projects and AI technologies, collaborating with partners across the industry, and developing solutions that empower organizations to flexibly deploy AI workloads wherever they need to be. Today, we announced that we have signed a definitive agreement to acquire Neural Magic. I believe this significant milestone will enable us to accelerate our progress and deliver on our vision for the future of AI.

At Red Hat, we believe the future of AI is open, and it hinges on several key pillars:

Small models unlock adoption

AI isn't just about massive, resource-intensive models. We're witnessing a shift towards smaller, more specialized models that deliver exceptional performance with greater efficiency. These models are not only more efficient to train and deploy, but they also offer significant advantages in terms of customization and adaptability.

Take, for example, IBM Granite 3.0, the third-generation of the Granite series LLMs, which emphasize smaller, functional AI models. Released under the permissive Apache 2.0 license, these models range in size from 1B to 8B parameters, allowing them to run anywhere from a laptop to standard GPU servers. Just as we saw with Linux, this ease of accessibility leads to innovation and adoption within the enterprise.

And beyond just a smaller starting size, optimizing AI models through sparsification and quantization is another force multiplier, allowing us to service more and more demand with the same hardware. Sparsification strategically removes unnecessary connections within a model drastically reducing its size and computational requirements without sacrificing accuracy or performance. Quantization then further reduces model size to run on platforms with reduced memory requirements. All of this translates to lower costs, faster inference and the ability to run AI workloads on a wider range of hardware. A similar focus on Linux has made it capable of running on practically any infrastructure on the planet - from watches to supercomputers. With Neural Magic joining Red Hat, we get to bring this same emphasis to the AI space.

Training unlocks business advantage

As powerful as these small AI models are, they are still trained on publicly accessible data. They have an incredible command of languages, they understand business, they know about most topics on the internet. But, almost by definition, they don't understand your business. If your business processes and your intellectual property aren’t in the public domain, it won’t understand them. And yet you need to refine your business processes, not the generic concept of one. So to truly unlock your business potential, you need to insert your knowledge into these models. And that requires training.

Red Hat is helping to enable this with InstructLab, an open source project designed to make it easier to contribute to and fine tune LLMs for gen AI applications, even by users who lack data science expertise. Launched by Red Hat and IBM and delivered as part of Red Hat AI, InstructLab is based on a process outlined in a research paper published in April 2024 by members of the MIT-IBM Watson AI Lab and IBM. This lowers the complexity to train an AI model for your needs, decidedly mitigating some of the most expensive aspects of enterprise AI and making LLMs more readily customizable for specific purposes.

Choice unlocks innovation

Most organizations have workloads that span both corporate datacenters and cloud infrastructure. AI should seamlessly integrate with existing infrastructure to enable flexible and consistent deployment across diverse environments, whether that’s on-premises, in the cloud or at the edge. You need to be able to train where you have your data and your resources. And you need to run your models wherever makes sense for your use case. Much like Red Hat Enterprise Linux (RHEL) allowed for applications compiled to it to run on any CPUs without changing the application, our mission is to make sure that models trained with RHEL AI can run on any GPU servers. This combination of flexible hardware, small models, simplified training and optimization provides the flexibility that will allow innovation to thrive.

We also believe that AI training and deployment at scale will require the same discipline that we have established in software over the last decade. Red Hat OpenShift AI brings the domains of model customization, inference, monitoring and lifecycle capabilities together with the applications consuming them on Red Hat OpenShift. Neural Magic shares the same passion for enabling AI to run across hybrid platforms and has shown leadership in the open source communities focused on driving innovation in this domain.

Amplifying the Neural Magic mission

Neural Magic was founded on the belief that AI should be able to run anywhere, from the smallest devices to the largest datacenters. Its origin story as a company parallels some of what I have seen in the small yet powerful teams at Red Hat that are innovating in AI, including our InstructLab team, so I think it is worth sharing here.

Nir Shavit, a renowned professor at MIT with a focus on parallel computing, had been exploring the intricacies of algorithms and hardware for decades. His work had already revolutionized areas like concurrent data structures and transactional memory. Alex Matveev, a former researcher at MIT, brought expertise in machine learning and a keen understanding of the challenges faced in deploying AI models efficiently.

The spark for Neural Magic ignited when Nir and Alex recognized a critical bottleneck in the advancement of AI: the dependence on expensive and often scarce GPUs. This reliance created a barrier to entry, hindering the widespread adoption of AI across various industries and limiting its potential to revolutionize how we live and work.

They embarked on a mission to empower anyone, regardless of their resources, to harness the power of AI. Their groundbreaking approach involved leveraging techniques like pruning and quantization to optimize machine learning models, starting by allowing ML models to run efficiently on readily available CPUs without sacrificing performance. Ultimately, Neural Magic shifted their vision to GPU acceleration and brought this same level of optimization and efficiency to gen AI through vLLM. This commitment to innovation promised to make AI more accessible, affordable, and easier to deploy. I am excited about the opportunity to bring those capabilities to our customers in Red Hat AI, but I am equally excited that our teams share a culture of experimentation and invention rooted in breaking through current limitations hindering AI’s progress and adoption.

In our Boston office, for example, you will find a driven group of passionate associates and researchers — coincidentally, from MIT — working on InstructLab to solve the training contribution bottleneck in widespread AI adoption. Just as Neural Magic’s technology is democratizing access to AI, InstructLab seeks to do the same for how we train and tune models. I can’t wait to see what additional breakthroughs this team can find, together.

I’m so excited about the prospect of Neural Magic joining Red Hat and accelerating our mission with the open source community to deliver on the future of AI. At Red Hat, we believe open unlocks the world’s potential. Our mission with Neural Magic will be to accelerate that unlocking with AI. I believe that doing this with the open source community will deliver the best outcome to the world.

We’re just getting started.