Blogs›Modal›Serverless GPU Inference
Modal's platform enables truly serverless GPU inference by optimizing replica spin-up times to tens of seconds. This is achieved through a combination of cloud buffers for pre-warmed GPUs, a custom lazy-loading filesystem, and advanced checkpoint/restore mechanisms for both CPU and CUDA contexts. Recent work has introduced GPU memory snapshotting, which can reduce median cold start times for certain workloads (like LLM inference with vLLM) by almost 10x, from minutes to seconds, by saving and re-using GPU memory. This post further elaborates on the economic advantages of serverless GPUs by introducing a cost model that compares serverless GPU pricing against reserved instances, highlighting that serverless GPUs are often cheaper when the peak-to-average demand ratio exceeds the reservation discount rate. It also discusses the assumptions and limitations of this cost model, including perfect demand prediction and instant allocation, and touches upon the developer productivity benefits of serverless GPU usage.
7/6/2026
Introduces a cost model to compare serverless GPU pricing with reserved instances, demonstrating that serverless GPUs are more cost-effective when the peak-to-average demand ratio is high. Discusses the assumptions and limitations of the cost model, including demand prediction and allocation speed, and highlights the developer productivity benefits of serverless GPUs.
5/12/2026
This post details the engineering efforts to achieve sub-50-second GPU inference replica spin-up times. Key contributions include: 1. Maintaining a buffer of healthy, idle GPUs to absorb immediate demand spikes. 2. Implementing a custom, content-addressed, multi-tier cloud-native filesystem for lazy loading of container images. 3. Developing checkpoint/restore capabilities for CPU processes to fast-forward initialization. 4. Extending checkpoint/restore to CUDA contexts for GPU-side initialization. These four components collectively reduce replica spin-up from tens of minutes to tens of seconds.
4/7/2026
Introduced availability of NVIDIA RTX Pro 6000 Blackwell GPUs, offering 96GB of VRAM and enhanced fp4/fp8 throughput for inference and fine-tuning workloads.
12/2/2025
Introduced GPU memory snapshotting as an experimental feature to drastically reduce cold start times for GPU-intensive workloads. This feature, when enabled with sleep mode for vLLM servers, shifts GPU memory to CPU memory, allowing for faster restoration from snapshots. Tested on Ministral 3 3B, achieving a 10x reduction in median cold start time.
11/19/2025
Reducto leveraged Modal's GPU memory snapshotting feature to reduce cold boot times for their inference models by 83%, from approximately 70 seconds to 12 seconds. This significantly improved their P90 latency by 3x for enterprise-scale document processing.
8/28/2025
This post details how Zencastr migrated their GPU-intensive audio transcription workloads from a self-managed Kubernetes cluster to Modal. Key technical contributions include: 1. Demonstrating the cost-effectiveness of Modal's scale-to-zero for spiky AI workloads compared to always-on Kubernetes GPU nodes. 2. Highlighting Modal Images for seamless management of diverse ML model dependencies and CUDA driver versions, reducing infrastructure management overhead. 3. Showcasing the flexibility of Modal for experimenting with different GPU models and concurrency settings without code changes. 4. Illustrating a large-scale batch audio processing architecture using Modal Functions, S3 Gateway endpoints, and Modal Volumes for efficient data handling and parallel GPU utilization (scaling to 1,500 concurrent GPUs).
7/30/2025
Introduced GPU memory snapshots, extending the existing memory snapshotting capability to include GPU state (device memory, CUDA kernels, streams, contexts). This is achieved by leveraging NVIDIA's CUDA checkpoint/restore APIs (`cuCheckpointProcessLock`, `cuCheckpointProcessCheckpoint`, `cuCheckpointProcessRestore`, `cuCheckpointProcessUnlock`). The implementation transparently captures and restores GPU memory, eliminating the need for manual GPU warm-up and recompilation after restore, leading to significantly faster cold starts for GPU-accelerated workloads.
5/30/2025
This post announces the availability of NVIDIA B200 and H200 GPUs on the Modal platform. It details the specifications of these new GPUs, comparing them to H100s, and explains how their increased memory, memory bandwidth, and FP4 Tensor Core support benefit large model inference, particularly for Mixture-of-Experts (MoE) models. Benchmarks are provided showing significant latency and throughput improvements for LLM inference (DeepSeek V3 Large) on B200s compared to H200s. The post also highlights Modal's infrastructure advantages for GPU deployment, including fast spin-up times, autoscaling, and pay-as-you-go pricing.
1/28/2025
Introduced memory snapshot restore functionality for serverless containers, leveraging gVisor's checkpoint/restore capabilities. This significantly reduces cold start latency by capturing and restoring the entire state of a Linux container, including its process tree and memory mappings, to disk. The system prioritizes loading memory pages in the background and aggressively preloads the snapshot 'pages' file into the host page cache to minimize page fault latency. This results in a 2.5x speedup for container startup, with examples showing a Stable Diffusion inference Function going from 13 seconds to 3.5 seconds, and an 'import torch' example from 5 seconds to ~1 second.
12/19/2024
Introduced support for NVIDIA L40S GPUs, offering 48GB of DDR6 RAM and significantly improved FP16/BF16 and FP8 Tensor Core arithmetic bandwidth compared to the A10 GPU. This enables running larger models and provides substantial speedups for both memory-bound and compute-bound inference workloads without requiring user tuning.