How we achieved truly serverless GPUs

What this post added

This post details the engineering efforts to achieve sub-50-second GPU inference replica spin-up times. Key contributions include: 1. Maintaining a buffer of healthy, idle GPUs to absorb immediate demand spikes. 2. Implementing a custom, content-addressed, multi-tier cloud-native filesystem for lazy loading of container images. 3. Developing checkpoint/restore capabilities for CPU processes to fast-forward initialization. 4. Extending checkpoint/restore to CUDA contexts for GPU-side initialization. These four components collectively reduce replica spin-up from tens of minutes to tens of seconds.