TechPrufer
Get started
Modal
Modal logoModal · Feature trail

Serverless GPU Inference

11 posts · Feb 2024 – Sep 2026

What this trail covers

Modal's platform enables truly serverless GPU inference by optimizing replica spin-up times to tens of seconds. This is achieved through a combination of cloud buffers for pre-warmed GPUs, a custom lazy-loading filesystem, and advanced checkpoint/restore mechanisms for both CPU and CUDA contexts. Recent work has introduced GPU memory snapshotting, which can reduce median cold start times for certain workloads (like LLM inference with vLLM) by almost 10x, from minutes to seconds, by saving and re-using model states. This post demonstrates how to leverage this infrastructure with smaller models and search to achieve frontier-level performance on code generation tasks, matching or exceeding GPT-4o's pass@1 performance on HumanEval with 100 LLaMA 3.1 8B generations.

All Modal trails
  1. How Botika runs full-stack generative AI on Modal | Modal BlogThis post details Botika's extensive use of Modal for their full-stack generative AI operations. It showcases how Modal's platform supports proprietary foundation models, large-scale data pipelines (100TB), and numerous production inference models across various GPU types (L4s, L40S, A100s, H100s). Specific technical aspects highlighted include: the ability to run complex data pipelines with thousands of concurrent containers, enabling rapid iteration for AI research through launching numerous experiments, multi-node training with cross-GPU communication and RDMA networking, and standing up reinforcement learning infrastructure quickly. The post emphasizes Modal's role in handling unpredictable AI traffic spikes through elastic scaling and the seamless integration of training, inference, and data pipeline components within a single platform.
  2. Kimi K3 by Moonshot now available on Modal | Modal BlogThis post announces the availability of Moonshot's Kimi K3 model on Modal, highlighting its 2.8 trillion parameters, 1M token context window, and native vision. It details Modal's partnership with Moonshot and vLLM to provide Day 0 support, including token-based pricing on the Shared API and dedicated Auto Endpoints. A custom-trained DFlash speculator, tuned to K3's architecture, is also introduced, achieving 360% faster interactivity and 88% higher throughput on agentic workloads. The post emphasizes the unglamorous but critical work involved in making such a large open model servable, including quantization-aware training and vLLM contributions.
  3. How to price serverless GPUsThis post introduces a cost model for pricing serverless GPUs versus reserved GPUs, focusing on the peak-to-average demand ratio as a critical factor. It provides equations and a conceptual framework for understanding when serverless GPUs are more economical. The post also discusses the assumptions and limitations of the model, including perfect prediction of demand and instant allocation/deallocation, and touches upon the developer productivity benefits of serverless GPU access.
  4. Multi-token Residual Prediction | Modal BlogThis post introduces Multi-Token Residual Prediction (MRP), a novel technique for accelerating diffusion language models (DLMs) during inference. Unlike previous Multi-Token Prediction (MTP) methods that predict the full next-step distribution, MRP trains a small auxiliary module to predict the residual between adjacent denoising steps. This residual prediction is a simpler target, allowing the module to be trained more effectively and applied across multiple steps without significant quality degradation. MRP can be used in two regimes: in the static regime, it enables (near-)lossless speedups via speculative decoding or direct decoding with a small quality cost; in the dynamic regime, it recovers accuracy lost to aggressive low-threshold decoding. The post details the implementation of MRP, its training objective, and its application in both speculative and direct decoding scenarios, demonstrating significant throughput improvements and quality recovery on various benchmarks.
  5. How we achieved truly serverless GPUsThis post details the engineering efforts to achieve truly serverless GPUs by reducing replica spin-up latency from tens of minutes to tens of seconds. It introduces four key optimizations: 1. Cloud buffers to maintain a pool of idle, healthy GPUs, removing instance allocation and health checks from the hot path. 2. A custom, content-addressed, multi-tier cloud-native cache for lazy loading of container images. 3. Checkpoint/restore mechanisms to fast-forward CPU-side initialization by directly restoring processes into memory. 4. CUDA checkpoint/restore to fast-forward GPU-side initialization by directly restoring CUDA contexts into memory. The post also discusses the importance of maximizing GPU Allocation Utilization for inference workloads and presents the optimization problem for managing instance allocation and buffers.
  7. Autoscaling Autoresearch: Give your agents elastic GPUs on Modal | Modal BlogThis post details how Modal's autoscaling capabilities provide elastic GPUs for AI research agents, enabling dynamic scaling from single instances to large clusters and back to zero. It demonstrates this with the 'autoresearch' agent on the Parameter Golf challenge, showcasing how the platform handles unpredictable research workloads by allowing agents to provision and release GPUs on demand. The post breaks down the timeline into stages: pipeline validation (single GPU), broad exploration (parallel single GPUs), validation at scale (multi-GPU), debugging (scaled down), and optimization (scaled up and down). It highlights the ease of scaling via parameter changes (e.g., `gpu='H100'` to `gpu='H100:8'`) and the resulting speedup and efficiency gains compared to fixed workstations or clusters. The agent's ability to adapt compute type (interactive sandbox vs. batch job) and scale based on task needs is a key contribution.
  8. Product Updates: RTX Pro 6000 Blackwell, Command K, Sandbox FS API and more | Modal BlogIntroduced availability of NVIDIA RTX Pro 6000 Blackwell GPUs, offering 96GB of VRAM and enhanced fp4/fp8 throughput for inference and fine-tuning workloads.
  9. Runway chooses Modal to power real-time inference for Runway Characters | Modal BlogThis post details how Runway leveraged Modal's infrastructure for real-time inference of Runway Characters. It highlights the need for sustained low latency, distributing inference across multiple GPUs with high-bandwidth communication, and Modal's capability to enable multi-node GPU clusters with RDMA networking. Runway was able to move from proof of concept to production in under 30 days by adding a single line of code on Modal to achieve this.
  10. Try GLM-5.1, the new frontier of open intelligence, on Modal | Modal BlogThis post details the deployment and usage of GLM-5, a large open-weights foundation language model, on Modal's infrastructure. It highlights the technical challenges of running such a large model, including its size (700GB in 8-bit precision) requiring multi-GPU deployment, and the specific inference strategies employed (tensor parallelism in MoE layers with DeepGemm kernels, data parallelism in attention layers with FlashMLA kernels). It also showcases integration with various AI frontends (Vercel AI SDK, OpenCode, OpenClaw, Claude Code) and provides reproducible deployment code. The post also announces a free endpoint for GLM-5 and discusses usage limits and production deployment options.
  11. Modal + Mistral 3: 10x faster cold starts with GPU snapshottingThis post introduces and details the implementation of GPU memory snapshotting as an alpha feature within Modal's serverless GPU inference platform. It demonstrates a nearly 10x reduction in median cold start times for Mistral 3 3B models served via vLLM, reducing it from approximately 118 seconds to 12 seconds. The mechanism involves enabling Sleep Mode for the vLLM server and setting `experimental_options={'enable_gpu_snapshot': True}` to shift GPU memory to CPU memory for snapshotting, enabling faster restoration on subsequent starts.
  12. How Reducto improved enterprise-scale document processing latency by 3xReducto achieved a 3x reduction in P90 latency for their enterprise-scale document processing workloads by migrating over 30 models to Modal. Key contributions include leveraging GPU memory snapshotting to reduce cold boot times by 83% (from ~70s to ~12s), enabling independent scaling of models by bundling them into single Modal Functions or keeping them separate, and using parameterized Functions to create independent autoscaling compute pools per customer. They also utilized granular regional control to pin latency-sensitive Functions to specific regions.
  13. Modal + Datalab: Deploy high-throughput document intelligence in <5 minutesThis post details the deployment of Datalab's Marker and Surya tools on Modal, showcasing a 10x throughput improvement per container for document intelligence tasks compared to local execution. It highlights the use of Modal Volumes for caching model weights to reduce cold start times and the platform's autoscaling capabilities for handling large batch jobs on hundreds of GPUs. The technical details include specifying GPU hardware (e.g., 'l40s'), using `modal.Image` for environment setup, and leveraging `modal.Volume` for model caching. It also mentions performance gains from hardware (H100 GPU), Flash Attention, and environment tuning (e.g., OMP_NUM_THREADS).
  14. How Zencastr transcribed hundreds of years worth of audio in just a few daysThis post details Zencastr's migration from managing their own Kubernetes GPU nodes to Modal for AI workloads, specifically speech transcription. Key contributions include: 1. Demonstrating cost savings through Modal's true scale-to-zero capability for spiky transcription workloads. 2. Highlighting seamless management of remote environments and dependency management via Modal Images and pre-installed CUDA drivers. 3. Showcasing flexibility in experimenting with different GPU models and concurrency settings for performance trade-offs. 4. Illustrating a large-scale batch audio processing architecture using Modal Functions, S3 Gateway endpoints, and Modal Volumes, culminating in a successful scale-up to 1,500 concurrent GPUs.
  15. GPU Memory Snapshots: Supercharging sub-second startupIntroduced GPU memory snapshots by leveraging NVIDIA's CUDA checkpoint/restore API. This allows for checkpointing and restoring GPU memory contents, CUDA kernels, CUDA objects (streams, contexts), and memory mappings. The process involves locking CUDA processes, checkpointing GPU memory and CUDA state to host memory, and then restoring it. This eliminates the need for separate CPU and GPU loading steps and allows compiled models and CUDA graphs to be restored directly, significantly reducing cold start times for GPU-accelerated workloads.
  16. Transcribe speech 100x faster and 100x cheaper with open modelsThis post details the implementation and optimization of a batch transcription service on Modal using NVIDIA's open ASR models (Parakeet and Canary). It compares the performance and cost against a proprietary API, demonstrating 100x faster or 100x cheaper transcription. Key technical contributions include: architecting a distributed batch processing system on Modal GPUs using the NeMo framework, optimizing data transfer and cold start times for end-to-end throughput, and comparing this to a CPU-based approach for saturating a proprietary API. The post highlights the engineering tradeoffs in balancing batching efficiency, memory packing, and time utilization for distributed workloads.
  17. Product updates: Multi-node training clusters, B200 and H200s, and Client 1.0 releaseIntroduced NVIDIA B200 and H200 GPUs for serverless inference, offering significant speedups. Enabled multi-node training clusters with the `@clustered` decorator, leveraging high-speed RDMA interconnect for linear scaling. Launched Modal Client 1.0, focusing on API stability and predictability, with updates including `modal.Volume.read_only`, `--secret` option for `modal shell`, timezone support for `Cron` schedules, and `--timestamps` flag for `modal app logs`.
  18. Introducing: B200s and H200s on ModalThis post announces the availability of NVIDIA B200 and H200 GPUs on Modal, detailing their specifications and performance benefits over H100s. It highlights increased GPU RAM (180GB for B200), higher memory bandwidth (8 TB/s for B200), and new FP4 Tensor Core arithmetic capabilities on the B200. Benchmarks show a 2.5x faster median time-to-first-token and 1.7x higher queries-per-second for DeepSeek V3 Large on 8xB200s vs 8xH200s using vLLM. The post emphasizes Modal's ability to provide these powerful GPUs with no reservations, fast spin-up times, and pay-as-you-go pricing.
  19. How sync. uses Modal to lipsync 100 hours of video a dayThis post details how sync. Labs leverages Modal's serverless GPU inference capabilities to process large volumes of video for lipsyncing. It describes their workflow of splitting videos into scenes, using T4 GPUs for face detection and translation, and A100 GPUs for the proprietary LipSync GenAI model. The post emphasizes Modal's ability to handle autoscaling and fast iteration, enabling sync. to deploy up to 95 times a day and ship 10 major model variants and 1000 iterations in one year. It highlights the efficient parallel execution model for compute-intensive workloads.
  20. Memory snapshots: Checkpoint/restore for sub-second startupThis post details the implementation of memory snapshot restores for serverless container startup. It explains how memory snapshots, derived from CRIU-like technology within gVisor, capture and restore the entire state of a Linux container's process tree. The primary benefit is a significant reduction in cold start latency, particularly for Python workloads, by avoiding thousands of filesystem operations and syscalls during program initialization. The performance gains are attributed to prioritized background page loading and aggressive preloading of pages into the host page cache. The post also discusses the lifecycle management of snapshots, including compatibility considerations with CPU featuresets, NVIDIA drivers, and container runtimes, and highlights the tradeoffs of increased system complexity.
  21. Introducing: L40S GPUs on ModalIntroduces support for NVIDIA L40S GPUs, which offer 48GB of VRAM (double the A10) and improved performance for inference workloads. The post details performance benefits for larger models and compute-bound jobs, including a 20% speedup in a basic load test without tuning. It also provides a comparative spec table and instructions for using the L40S with the Modal Python SDK.
  22. How OpenArt scaled their Gen AI art platform on hundreds of GPUsThis post details how OpenArt integrated their custom ComfyUI workflows into Modal for Gen AI art generation. They overcame inflexibility of model API providers and slow deployment cycles on GCP by defining container images with custom nodes, deploying functions with GPU requirements and file mounts, and leveraging Modal's autoscaling to handle hundreds of GPU containers. This enabled them to deploy in minutes instead of hours and scale their platform to millions of users.
  23. The future of AI needs more flexible GPU capacityThis post argues for the need for more flexible GPU capacity, particularly for inference workloads, due to their inherent volatility. It details how demand pooling, supply pooling across regions and cloud providers, fast scaling (seconds vs. minutes), and demand smoothing can improve GPU utilization. The author highlights Modal's investment in resource pool scaling, bin packing using mixed-integer programming, and serverless infrastructure with fast model initialization (seconds or less) through custom file systems and memory snapshotting.
  24. Beat GPT-4o at Python by searching with 100 dumb LLaMAsThis post demonstrates a novel application of Modal's serverless GPU inference capabilities by combining smaller LLaMA 3.1 8B models with extensive search to achieve performance comparable to or exceeding GPT-4o on the HumanEval benchmark. It details the use of vLLM for efficient LLM inference, Modal's sandboxing technology for secure and parallelized code evaluation, and presents scaling laws for 'fail@k' metrics, showing predictable performance improvements with increased compute. The experiment highlights the trade-off between model size and search compute, and the cost-effectiveness of this approach on Modal's platform.
  25. How Suno shaved 4 months off their launch timeline with ModalThis post details how Suno used Modal's serverless GPU inference and batch pre-processing capabilities to accelerate their product launch. It emphasizes Modal's ability to dynamically scale compute resources (up to thousands of GPUs) to meet variable demand, avoiding the need for Suno to manage their own infrastructure or commit to long-term GPU reservations. The post highlights the ease of use, with Python scripts defining workflows, and the benefit of avoiding complex Kubernetes setups.
  26. Introducing: H100s on ModalIntroduces direct access to NVIDIA H100 GPUs, offering significantly higher performance for training and inference compared to A100s. Highlights H100 specifications (80GB HBM2e, NVLink) and pricing ($7.65/hour/GPU). Provides a code example for specifying 'H100' in the Modal decorator. Recommends H100s for latency-sensitive applications and throughput-bound jobs like fine-tuning, emphasizing Modal's pay-per-use model and autoscaling for cost-effectiveness.