GPU Memory Snapshots: Supercharging sub-second startup
Jul 30, 2025

Summary
Introduced GPU memory snapshots by leveraging NVIDIA's CUDA checkpoint/restore API. This allows for checkpointing and restoring GPU memory contents, CUDA kernels, CUDA objects (streams, contexts), and memory mappings. The process involves locking CUDA processes, checkpointing GPU memory and CUDA state to host memory, and then restoring it. This eliminates the need for separate CPU and GPU loading steps and allows compiled models and CUDA graphs to be restored directly, significantly reducing cold start times for GPU-accelerated workloads.