Introduction to llm-d

This experience highlights:

• How LLM-D separates prefill and decode stages to scale each independently
• Testing prompts and seeing cache hit rates in action
• Comparing repeated and unique prompts to show the impact on latency and load distribution
• Using the inference gateway for load testing and measuring session stickiness
• Tracking per-pod cache hit rates, token throughput, and latency in Grafana

Red Hat LLM-D helps scale distributed inference across environments, improving efficiency and reducing latency for generative AI applications.