llm-d: Distributed LLM Inference on Kubernetes
Shriira Press
Serving large language models at scale on Kubernetes — disaggregated, cache-aware, and built on vLLM.
Welcome to llm-d: Distributed LLM Inference on Kubernetes.
llm-d is a Kubernetes-native framework for serving large language models efficiently at production scale. A model server like vLLM can run a model fast on a single accelerator, but real traffic is messy: prompts share long prefixes, some requests need huge context windows while others want quick replies, and a fleet of GPUs has to stay busy without blowing past latency targets. llm-d sits above the model server and solves that higher-level problem, weaving together intelligent request routing, KV-cache-aware scheduling, prefill/decode disaggregation, and wide expert parallelism into recipes the project calls well-lit paths. This book begins with the inference-at-scale problem and the three pillars llm-d builds on — vLLM, Kubernetes, and the Gateway API Inference Extension. From there it works through the inference scheduler and its Endpoint Picker, the KV-cache indexer that tracks where context already lives, the splitting of prefill from decode, and the wide expert parallelism that makes massive Mixture-of-Experts models practical. It closes with deployment, autoscaling, operations, and where llm-d fits in the cloud-native ecosystem.
This title is part of the ShriIra library and is free to read in full, right here — our small contribution to making world-class knowledge easy to reach.
A note on reading it: open the Contents menu at the top of the reader to jump between chapters, use the Aa menu to set a comfortable text size, theme (light, sepia, or night), and single- or two-page layout. Your place is saved automatically, so you can always pick up where you left off.
We hope it serves you well.
— Shriira Press