Inference Performance Optimization

6/23/2026

Introduces Modal Auto Endpoints, a self-serve system for deploying production-grade LLM inference services. This feature provides users with transparent access to the underlying code, inference engine configurations (e.g., DFlash, SGLang, FlashAttention-4), and detailed engine-level observability metrics (TTFT, ITL, speculative decoding acceptance length). It leverages Modal's existing AI infrastructure, including serverless GPU inference, autoscaling, and Modal Servers for low-latency routing. The system is built on recipes derived from internal optimization efforts and offers a declarative interface based on workloads and SLOs, with an evolving agentic system for generating deployment configurations.

5/4/2026

This post details an optimization in SGLang's scheduler to improve multimodal inference performance. By profiling the scheduler with `py-spy`, a bottleneck was identified in the `process_input_requests` function, specifically within calls to `torch.UntypedStorage._new_shared_cuda`. This was caused by repeatedly reopening shared CUDA IPC handles for tensors. The solution implemented a Python dictionary cache (`_pool_storage_cache`) to store and reuse these handles, significantly reducing host-side overhead. This optimization resulted in a ~16% increase in throughput and a ~10% reduction in mean end-to-end latency for multimodal workloads, and also improved decode latency by reducing overall scheduler contention.

4/8/2026

Introduced a specialized QUIC-based transport over UDP with automatic NAT traversal for real-time robotic inference. This system establishes a persistent, bidirectional channel between the robot runtime and the Modal GPU container, streaming observations and receiving action outputs over the same channel. It handles automatic NAT traversal using STUN discovery and UDP hole punching, coordinated via Modal Dict for rendezvous. This reduces network overhead to ~10-15ms and avoids potential request-response stalls in control loops.

3/26/2026

This post details how Runway leverages Modal's platform for real-time inference of Runway Characters, a video agent API. It highlights Modal's ability to handle GPU-intensive, latency-critical, and variable demand workloads. Specifically, it mentions Runway's ability to turn containers into multi-node GPU clusters with RDMA networking via a single line of code, enabling distribution of inference across multiple GPUs with high-bandwidth communication between nodes. This allows for sustained low latency across the full duration of a conversation, with expressions, lip-sync, and gestures, without degradation, and deployment across global regions at production scale.

3/25/2026

This post details how Doppel migrated their ML training and inference workflows to Modal, significantly reducing ML infrastructure tax. For training, it highlights the use of Modal's `map()` function to parallelize K-fold cross-validation and other experiments, shortening feedback loops. For inference, it describes how Modal simplifies deployment by removing the need for manual HTTP service layers, reducing build times through image layer caching and persistent volumes, and automating scaling for traffic spikes. The post contrasts the previous GCP inference stack with the Modal approach, emphasizing faster iteration and reduced operational overhead.

11/18/2025

This post details how Modal addresses host overhead in AI inference, a key contributor to GPU inefficiency. It explains how to identify host overhead by monitoring GPU kernel utilization and discusses strategies to mitigate it, including avoiding unnecessary CPU-GPU data transfers (e.g., constructing tensors on the GPU instead of transferring them), fusing multiple kernels into a single launch to reduce kernel launch overhead, and utilizing CUDA Graphs to amortize launch overhead further. The post highlights contributions to open-source inference engines like SGLang and vLLM to implement these optimizations.

10/29/2025

This post details the deployment of Datalab's Marker and Surya document intelligence models on Modal. It highlights how Modal's platform, specifically its GPU compute capabilities, autoscaling, and caching mechanisms (Modal Volumes), enables high-throughput document processing. The post provides a quickstart guide and explains the underlying technical mechanisms, including environment specification via decorators, GPU usage (e.g., L40s), and the benefits of Modal Volumes for caching model weights. It also quantifies performance gains, showing a 10x throughput increase for Marker on Modal compared to local CPU processing, and discusses how Modal's autoscaling can further improve batch processing throughput.

7/23/2025

This post details the implementation of a high-throughput, low-cost batch transcription service on Modal using open-source ASR models (NVIDIA's Parakeet and Canary). It describes architectural choices for distributing transcription requests across Modal GPUs, including batching strategies and end-to-end performance measurement (throughput, cold starts, data transfer) to compare against proprietary APIs. The work demonstrates achieving over 100x faster or cheaper transcription by leveraging Modal's autoscaling GPU infrastructure and open models.

7/16/2025

This post introduces a new perspective on cost and performance optimization for LLM inference, advocating for a 'dollars per request' model instead of 'dollars per token'. It explains why this shift is crucial for teams self-hosting inference, impacting latency estimation, replica scaling, and overall cost analysis. The post highlights how this framing aligns engineering efforts with product and business goals, and suggests Modal as a platform to achieve lower costs per request.

6/18/2025

This post details optimizations applied to the FLUX.1-dev diffusion model for faster inference on Modal. It introduces standard optimizations including using the Torch compiler (`torch.compile` with `max-autotune`), fusing QKV projections, and adopting the channels-last memory layout, which collectively achieved a 1.5x speedup. Additionally, it describes the implementation of an approximate caching technique called First Block Caching, adapted from the ParaAttention repo and TEACache paper, which further reduced latency by 2x, leading to a total 3x performance improvement. The post also touches on reducing compile-time costs for `max-autotune`.

4/24/2025

This post details how Lemon Slice built a real-time generative video product using Modal and Daily. Key technical contributions include: utilizing Modal Functions for scalable inference of a 1B parameter video model, achieving autoscaling for GPU containers based on request volume, leveraging memory snapshotting for faster container initialization, and implementing a low-latency architecture for real-time video conversations. This involved co-locating inference containers using region selection and enabling direct communication via Modal's Tunnel feature to minimize latency between containers processing user input and generating video.

4/18/2025

This post details how sync. uses Modal's parallel execution model to process over 100 hours of video per day for lipsyncing. The workflow involves splitting videos into scenes, running face detection and translation on T4 GPU containers, processing scenes with proprietary LipSync GenAI models on A100 GPUs, and then stitching the dubbed scenes back together. This demonstrates the platform's capability for large-scale, compute-intensive video processing workloads.