Modal

GPU Memory Snapshots: Supercharging sub-second startup

Jul 30, 2025
GPU Memory Snapshots: Supercharging sub-second startup
SummaryAI, from the source post

Introduced GPU memory snapshots by leveraging NVIDIA's CUDA checkpoint/restore API. This allows for checkpointing and restoring GPU memory contents, CUDA kernels, CUDA objects (streams, contexts), and memory mappings. The process involves locking CUDA processes, checkpointing GPU memory and CUDA state to host memory, and then restoring it. This eliminates the need for separate CPU and GPU loading steps and allows compiled models and CUDA graphs to be restored directly, significantly reducing cold start times for GPU-accelerated workloads.

More from the Serverless GPU Inference trail