Technology · Ebook

Vineyard: Zero-Copy In-Memory Data Sharing for Big Data and ML

by Shriira Press

Vineyard, known in code by its shorthand v6d, is a CNCF sandbox project that solves a quietly expensive problem: when one stage of a data pipeline hands intermediate results to the next, it usually serializes them to a file or object store and the receiver deserializes them back, and for large dataframes, tensors, and graphs that round trip dominates the runtime. Vineyard is an in-memory immutable data manager that lets co-located stages share data directly through shared memory, zero-copy and serialization-free. This book moves from the problem to Vineyard's metadata-and-blob object model, the vineyardd daemon, IPC and RPC clients, distributed and global objects, the builder and resolver registry, the Kubernetes operator and data-aware scheduler, and the integrations and practices that make it pay off.

Contents

1. 1Preface
2. 2Chapter 1 — The Cost of Moving Data
3. 3Chapter 2 — The Object Model
4. 4Chapter 3 — Inside vineyardd
5. 5Chapter 4 — Accessing Data: IPC and RPC
6. 6Chapter 5 — Builders, Resolvers, and Drivers
7. 7Chapter 6 — Distributed and Global Objects
8. 8Chapter 7 — Vineyard on Kubernetes
9. 9Chapter 8 — Vineyard in Practice