vLLM roundup April 2025

We are heading to NYC for our next vLLM Meetup on Wednesday, May 7th at the IBM Office (1 Madison Avenue). Hosted by Red Hat and IBM, this in-person gathering will feature deep dives and lightning talks from experts at AMD, IBM, Meta’s PyTorch Team, and the vLLM crew from Red Hat. Spots are limited, and registration approval is required, so make sure to request to join here before it fills up!

We hope to see you there!

Bi-weekly vLLM Office Hours

Upcoming

vLLM Office Hours #25: Structured Outputs in vLLM

May 8, 2025 - 2:00PM ET / 11:00AM PT

Register Here

vLLM Office Hours #26: Intro to torch.compile and How It Works with vLLM

May 29, 2025 - 2:00PM ET / 11:00AM PT

Register Here

Recordings you don't want to miss 

Performance Optimization of vLLM on Google TPUs | Video

Deep Dive Into the LLM Compressor | Video

Intro to vLLM V1 | Video

Red Hat AI Innovation team: Friday Random Samples weekly series

No Math AI podcast

Generative Optimization in Engineering Design | Watch Here

Inference-time scaling: How small models beat the big ones | Watch Here

AI blog highlights 

Cracking the code: How neural networks might actually “think”

Deep neural networks are achieving the incredible, pushing the boundaries of artificial intelligence in areas from medicine to language. But as these powerful AI systems become more integrated into our lives, a critical challenge looms: we often don’t understand how they arrive at their answers. They operate like inscrutable “black boxes,” making it hard to fully trust them. 

Keep Reading

Performance boosts in vLLM 0.8.1: Switching to the V1 engine

vLLM has rapidly become the go-to solution for efficient inference of large language and multimodal models. In this post, we'll demonstrate the substantial performance and usability improvements introduced in vLLM 0.8.1 compared to version 0.7.3, emphasizing crucial architectural overhauls and multimodal inference optimizations.

Keep Reading 

Transformers backend integration in vLLM

The Hugging Face Transformers library offers a flexible, unified interface to a vast ecosystem of model architectures. From research to fine-tuning on custom dataset, transformers is the go-to toolkit for all.

But when it comes to deploying these models at scale, inference speed and efficiency often take center stage. Enter vLLM, a library engineered for high-throughput inference, pulling models from the Hugging Face Hub and optimizing them for production-ready performance.

Keep Reading 

Accelerating RLHF with vLLM, Best Practice from OpenRLHF

As demand grows for training reasoning-capable large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) has emerged as a cornerstone technique. However, conventional RLHF pipelines—especially those using Proximal Policy Optimization (PPO)—are often hindered by substantial computational overhead. 

Keep Reading 

AI research from our lab

We recently launched AI Research Hub, a destination for all research from Red Hat and Neural Magic labs. We plan to post all our research papers, research blogs, and accompanying code to this new location, so please bookmark it! Here are three papers we are currently featuring on the new page:

• Towards Combinatorial Interpretability of Neural Computation 
  Paper Link | Blog
• LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation 
  Paper Link | Code
• Unveiling the Secret Recipe: A Guide for Supervised Fine-Tuning Small LLMs 
  Paper Link | Code
• Implicit In-Context Learning 
  Paper Link | Code
• A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods 
  Paper Link | Code
• Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning 
  Paper Link
• SQuat: Subspace-Orthogonal KV Cache Quantization 
  Paper Link
• Activation-Informed Merging of Large Language Models 
  Paper Link

Stay engaged with the vLLM community

• vLLM Discord
• Developer Slack
• vLLM Blog

vLLM is nearing 46,000 stars! 🌟 Be sure to add your star and join the community. Thank you for your support.