TechPrufer
Sign inGet started

Blogs›NVIDIA›Fast Startup for Inference Workloads on Kubernetes

Fast Startup for Inference Workloads on Kubernetes

Fast Startup for Inference Workloads on Kubernetes

1
posts
2026

This feature thread tracks the evolution of enabling fast startup for AI inference workloads on Kubernetes. Initial efforts focused on the cold-start problem, where initializing inference replicas can take several minutes, leading to idle GPUs and potential SLA violations during traffic spikes. NVIDIA Dynamo Snapshot addresses this by leveraging CRIU and cuda-checkpoint for near-instant checkpoint/restore of single-GPU AI inference workloads. Key optimizations include quiesce/resume hooks for workload cleanup and resource recreation, KV cache unmapping to reduce checkpoint size, parallel memfd restore and AIO for faster memory restoration, and a GPU Memory Service (GMS) with GPUDirect Storage for efficient model weight handling. Experimental results show significant startup time reductions, paving the way for multi-GPU/multi-node support and TensorRT-LLM integration.

2026

NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes | NVIDIA Technical Blog

5/28/2026

Introduces NVIDIA Dynamo Snapshot, a checkpoint/restore approach for AI inference workloads on Kubernetes. It details the use of CRIU and cuda-checkpoint to serialize and restore both host and GPU device states, enabling near-instant startup. The post describes the Kubernetes integration via a snapshot-agent DaemonSet and workload-level quiesce/resume hooks. It highlights optimizations like KV cache unmapping, parallel memfd restore, AIO, and a GPU Memory Service (GMS) with GPUDirect Storage for reducing checkpoint size and accelerating restoration. Experimental results demonstrate significant startup time reductions for large models.