Serverless GPU Inference
Memory snapshots: Checkpoint/restore for sub-second startup

Memory snapshots: Checkpoint/restore for sub-second startup

1/28/2025

What this post added

Introduced memory snapshot restore functionality for serverless containers, leveraging gVisor's checkpoint/restore capabilities. This significantly reduces cold start latency by capturing and restoring the entire state of a Linux container, including its process tree and memory mappings, to disk. The system prioritizes loading memory pages in the background and aggressively preloads the snapshot 'pages' file into the host page cache to minimize page fault latency. This results in a 2.5x speedup for container startup, with examples showing a Stable Diffusion inference Function going from 13 seconds to 3.5 seconds, and an 'import torch' example from 5 seconds to ~1 second.

Read the original post ↗