TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Fast Startup for Inference Workloads on Kubernetes

2 posts · May 2026 – Aug 2026

What this trail covers

This feature thread tracks the evolution of enabling fast startup for AI inference workloads on Kubernetes. Initial efforts focused on the cold-start problem, where initializing inference replicas can take several minutes, leading to idle GPUs and potential SLA violations during traffic spikes. NVIDIA Dynamo Snapshot addresses this by leveraging CRIU and cuda-checkpoint for near-instant checkpoint/restore of single-GPU AI inference workloads. Key optimizations include quiesce/resume hooks for workloads, KV cache unmapping, parallel memfd restore, Linux native AIO, and a GPU Memory Service (GMS) for decoupling model weights. Experimental results show significant startup time reductions for large models.

All NVIDIA trails
  1. Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo | NVIDIA Technical BlogIntroduces Shadow Engine Recovery in NVIDIA Dynamo, a feature that keeps a fully initialized shadow engine idle on the same GPUs as the active engine. Leverages the GPU Memory Service (GMS) to share weights without duplicating them in HBM, enabling recovery within seconds. Demonstrates a 39x speedup in recovery time for a GLM-5.2 deployment compared to a cold restart. Details the GMS architecture using CUDA Virtual Memory Management API and its integration with inference frameworks like vLLM. Explains the concept of parked shadow engines that defer KV cache materialization.
  2. NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes | NVIDIA Technical BlogThis post introduces NVIDIA Dynamo Snapshot, a checkpoint/restore approach for AI inference workloads on Kubernetes. It details the use of CRIU for host state and cuda-checkpoint for device state serialization. The implementation leverages a privileged DaemonSet (`snapshot-agent`) for container-level checkpointing and restore, and introduces quiesce/resume hooks within the inference worker to manage engine initialization and distributed runtime startup. Key optimizations for reducing checkpoint size and accelerating restore include KV cache unmapping, parallel memfd restore, Linux native AIO, and a GPU Memory Service (GMS) that decouples large model weights from process state for concurrent restoration using high-bandwidth channels like GPUDirect Storage. Experimental results demonstrate up to 21x startup time reduction on large models.