Fast, efficient inference with Red Hat AI

vLLM  is an inference engine designed to maximize throughput and accelerate response times across hardware accelerators. It uses the PagedAttention algorithm to optimize GPU utilization and speed up the output of generative AI applications. 

Use vLLM to optimize the deployment of any gen AI model, on any AI accelerator, while maintaining controlled and predictable inference behavior in production environments.

llm-d is a Kubernetes-native, open source framework that speeds up distributed LLM inference at scale. 

This means when an AI model receives complicated queries with a lot of data, llm-d provides a framework that makes processing faster. Its accessible, modular architecture makes llm-d an ideal platform for distributed LLM inference at scale. This supports scalable inference while maintaining consistency, control, and governance across distributed workloads.

Use metrics and insights of models in production to find out where and how your models can improve. See model-specific performance metrics like Time-to-First Token, KV-cache hit rate, and GPU utilization. Use these metrics to monitor performance, detect anomalies, and help inference meet operational, security, and policy requirements. 

Optimize your choice of foundational or custom models with a diverse model toolkit. Use techniques like quantization or sparsity to reduce hardware requirements and lower inference costs. 

Tools like LLM Compressor are included in the toolkit. It uses the latest model compression research to make LLMs smaller, more energy efficient, and faster. This reduces hardware requirements and improves efficiency—without sacrificing accuracy.

Beyond its core functionality, LLM Compressor integrates broadly with other tools and platforms. It supports inference within the Hugging Face Transformers ecosystem, enabling accuracy validation pre-deployment. It also interfaces with fine-tuning frameworks, allowing users to maintain sparsity during supervised training.

It helps achieve all of the above while maintaining validation, reproducibility, and control over model behavior before deployment.

Learn more about LLM Compressor

Red Hat AI includes the integration of a managed API gateway that allows AI platform engineers to set up internal models-as-a-service (MaaS) capabilities. It provides an open, modular, and vendor-neutral way to deploy and operate models across hybrid cloud environments.

Governed access to models through a centralized MaaS architecture lets you control who can access specific models, enforce policies, and monitor usage across users, applications, and agents. This supports reliable, auditable, and policy-driven model consumption at scale.

With easier ways to consume AI models and GPU resources, developers can streamline access to API end points and platform engineers can control, govern, and monitor access consumption for their high-performing, self-hosted models.

When paired with an inference stack that supports the unpredictable demand and scale of models and agents, an open strategy to manage model access provides a strong foundation for agentic AI, fine-tuning, and AI at scale.

Use any gen AI model or choose from our optimized collection of open source, third-party models, validated to run efficiently across the Red Hat AI platform.

Red Hat AI model validation is done using open-source tooling such as GuideLLM, Language Model Evaluation Harness, and vLLM. This supports reproducibility for customers and ensures models are validated, trusted, and consistently deployed across environments.