KAI Scheduler: Fair GPU Scheduling for AI on Kubernetes
Shriira Press
How NVIDIA's KAI Scheduler shares scarce GPUs fairly across teams, schedules distributed AI jobs as a gang, and packs clusters tight enough to keep every accelerator earning its keep.
Welcome to KAI Scheduler: Fair GPU Scheduling for AI on Kubernetes.
KAI Scheduler is a Kubernetes-native scheduler, open-sourced by NVIDIA from the Run:ai platform, that exists to solve one stubborn problem: GPUs are expensive, scarce, and badly served by the default Kubernetes scheduler, which was designed for stateless web pods rather than distributed training runs that must launch every worker at once. This book walks through how KAI fixes that. We begin with why AI workloads break ordinary scheduling and what KAI is, then look at its operator-driven architecture and the scheduling cycle that runs every second. From there we examine hierarchical queues and the fair-share machinery that divides capacity across teams, gang scheduling and the PodGroup that makes distributed jobs atomic, and the GPU-sharing and fractional-GPU features that let several small workloads occupy a single card. Later chapters cover priorities, preemption, reclaim, bin-packing and consolidation, plus topology-aware placement; a closing chapter ties it together with installation, queue setup, framework integrations, and operational practice. Throughout, the focus is on real components, real CRDs, and how the pieces actually fit.
This title is part of the ShriIra library and is free to read in full, right here — our small contribution to making world-class knowledge easy to reach.
A note on reading it: open the Contents menu at the top of the reader to jump between chapters, use the Aa menu to set a comfortable text size, theme (light, sepia, or night), and single- or two-page layout. Your place is saved automatically, so you can always pick up where you left off.
We hope it serves you well.
— Shriira Press