NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes | NVIDIA Technical Blog

What this post added

Introduces NVIDIA Dynamo Snapshot, a checkpoint/restore approach for AI inference workloads on Kubernetes. It details the use of CRIU and cuda-checkpoint to serialize and restore both host and GPU device states, enabling near-instant startup. The post describes the Kubernetes integration via a snapshot-agent DaemonSet and workload-level quiesce/resume hooks. It highlights optimizations like KV cache unmapping, parallel memfd restore, AIO, and a GPU Memory Service (GMS) with GPUDirect Storage for reducing checkpoint size and accelerating restoration. Experimental results demonstrate significant startup time reductions for large models.