How we achieved truly serverless GPUs

This post details the engineering efforts to achieve truly serverless GPUs by reducing replica spin-up latency from tens of minutes to tens of seconds. It introduces four key optimizations: 1. Cloud buffers to maintain a pool of idle, healthy GPUs, removing instance allocation and health checks from the hot path. 2. A custom, content-addressed, multi-tier cloud-native cache for lazy loading of container images. 3. Checkpoint/restore mechanisms to fast-forward CPU-side initialization by directly restoring processes into memory. 4. CUDA checkpoint/restore to fast-forward GPU-side initialization by directly restoring CUDA contexts into memory. The post also discusses the importance of maximizing GPU Allocation Utilization for inference workloads and presents the optimization problem for managing instance allocation and buffers.