Technology · Ebook

llm-d: Distributed LLM Inference on Kubernetes

by Shriira Press

llm-d is a Kubernetes-native framework for serving large language models efficiently at production scale. A model server like vLLM runs a model fast on one accelerator, but real traffic is messy: prompts share long prefixes, contexts vary wildly, and a fleet of GPUs must stay busy without missing latency targets. llm-d sits above the engine, weaving together intelligent request routing, KV-cache-aware scheduling, prefill/decode disaggregation, and wide expert parallelism into repeatable well-lit paths. This book starts with the inference-at-scale problem and the three pillars llm-d builds on — vLLM, Kubernetes, and the Gateway API Inference Extension — then works through the inference scheduler, the KV-cache indexer, disaggregated serving, expert parallelism, and finally deployment, autoscaling, and where llm-d fits in the cloud-native ecosystem.

Contents

1. 1Preface
2. 2Chapter 1 — The Problem of Serving LLMs at Scale
3. 3Chapter 2 — Foundations: vLLM, Kubernetes, and the Inference Gateway
4. 4Chapter 3 — Intelligent Inference Scheduling
5. 5Chapter 4 — KV-Cache-Aware Routing
6. 6Chapter 5 — Prefill/Decode Disaggregation
7. 7Chapter 6 — Wide Expert Parallelism
8. 8Chapter 7 — Deploying and Operating llm-d
9. 9Chapter 8 — llm-d in Practice