Technology · Ebook

KAITO: The Kubernetes AI Toolchain Operator

by Shriira Press

KAITO is a CNCF sandbox operator suite that automates large language model inference, fine-tuning, and retrieval-augmented generation on Kubernetes. Running an LLM normally means wrestling with GPU node pools, distributed-inference flags, multi-gigabyte weights, and a serving engine; KAITO replaces all of it with one custom resource. You name a model and a GPU type, and the operator estimates memory, provisions nodes, loads weights, and serves an OpenAI-compatible vLLM endpoint. This book covers the Workspace resource, the controller-and-provisioner architecture, GPU auto-provisioning via Karpenter, vLLM and preset models, LoRA and QLoRA fine-tuning, the RAGEngine, and scaling with InferenceSet and InferencePool.

Contents

1. 1Preface
2. 2Chapter 1 — What KAITO Is
3. 3Chapter 2 — The Workspace Custom Resource
4. 4Chapter 3 — Architecture and Controllers
5. 5Chapter 4 — GPU Node Auto-Provisioning
6. 6Chapter 5 — Inference with vLLM and Preset Models
7. 7Chapter 6 — Fine-Tuning with LoRA and QLoRA
8. 8Chapter 7 — Retrieval-Augmented Generation
9. 9Chapter 8 — Scaling and Routing Inference
10. 10Chapter 9 — KAITO in Practice