TechPrufer
Get started
Modal
Modal logoModal · Feature trail

Inference Performance Optimization

20 posts · Jan 2024 – Oct 2026

What this trail covers

Modal's platform enables significant performance improvements for AI inference through various optimizations. This includes reducing host overhead in schedulers, optimizing multimodal inference by caching shared GPU memory handles, enhancing speculative decoding for LLMs, enabling serverless GPU inference with rapid spin-up times, and supporting real-time generative video by co-locating inference containers and using direct communication via Tunnels. Recent work has focused on optimizing diffusion models and scaling to trillions of tokens for System One models like Jev, handling bursty, high-concurrency workloads with ultra-low-latency inference without managing infrastructure.

All Modal trails
  1. TypeSafe AI scales Jev to trillions of tokens in three days on Modal | Modal BlogThis post details how TypeSafe AI scaled their System One Model, Jev, to trillions of tokens in three days on Modal. It highlights Modal's ability to handle bursty, high-concurrency workloads with ultra-low-latency inference, enabling TypeSafe to focus on model development rather than infrastructure management. The post emphasizes Modal's suitability for new classes of frontier models beyond traditional LLMs, specifically mentioning Jev's parallel sampler and RLCD training method, and its output of typed, structured values with probabilities.
  2. Modal Clusters are generally availableIntroduces Modal Clusters, a new primitive for multi-node workloads, accessible via the `@modal.clustered` decorator. This enables communication between nodes via InfiniBand verbs at up to 6.4 Tbps, automatically configured for PyTorch and NCCL. The post details the implementation of a new gang scheduler with an observe-plan-act loop for atomic N-node placement. It also describes the integration of RDMA support into gVisor, including upstreaming changes to the OSS project, to enable secure multi-tenant RDMA communication. The post highlights customer use cases for training and inference at scale.
  3. Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine | Modal BlogIntroduced Quail, a query-aware inference layer that integrates a query planner with an inference engine. Achieved over 1 billion tokens per minute per H100 GPU, outperforming vLLM by 10x on specific AI-SQL benchmarks. The core innovation lies in leveraging query plans to optimize KV cache usage and reduce host overhead, enabling more efficient processing of large-scale, structured inference workloads.
  4. How to serve trillions of tokens for trillion-parameter coding agents | Modal BlogThis post details Modal's optimizations for serving trillion-parameter coding agents, focusing on achieving high throughput and interactivity for LLM inference. Key contributions include: optimizing inference performance by 2.8x per user and 5.6x across users on the replica; detailing the workload structure of coding agents, which involves long input sequences relative to output sequences due to iterative session history accumulation; and discussing the use of Moonshot AI's Kimi K2.6 model, its trillion parameters, and the NVFP4 micro-scaling format for hardware acceleration on Blackwell GPUs. The post also highlights upstreamed patches to the SGLang inference engine to improve prefill and decode performance, and discusses the trade-offs between interactivity and throughput in serving such models at scale.
  5. Kimi K3 by Moonshot now available on Modal | Modal BlogThis post details the integration of Kimi K3, a 2.8 trillion parameter multimodal model with a 1M token context window, on Modal. It highlights achieving 460 tokens per second on release day through partnership with Moonshot and vLLM. Key optimizations include Day 0 support for a custom DFlash speculator tuned to K3's architecture, resulting in 360% faster interactivity and 88% higher throughput on agentic workloads. The post also touches on the underlying architecture of K3, including its mixture-of-experts design, Kimi Delta Attention, and Attention Residuals, and the engineering effort involved in making such a large model servable.
  7. Inkling by Thinking Machines now available on Modal | Modal BlogIntroduced support for Thinking Machines' Inkling model, a 975B parameter MoE transformer with a 1M token context window and native audio/vision. Integrated a custom DFlash speculator tuned for Inkling's local attention architecture, achieving 250 tokens/sec/user at 2.5M TPM on 8x B200s, a 67% improvement over the model's built-in speculative path. Detailed DFlash's block diffusion drafting and its adaptation for local attention, including causal layers for optimized kernel support.
  8. Multi-token Residual Prediction | Modal BlogThis post introduces Multi-Token Residual Prediction (MRP) for diffusion language models (DLMs). MRP trains a small module to predict the residual between adjacent denoising steps, rather than the full next-step distribution. This approach allows for either lossless speedups in static denoising (via speculative decoding) or quality recovery in dynamic denoising regimes. The post details the failure of a naive MTP approach and the success of the residual prediction method, demonstrating its application in both speculative and direct decoding scenarios with significant throughput gains and minimal quality degradation on various benchmarks.
  9. Introducing Modal Auto Endpoints: Optimized inference you actually own | Modal BlogIntroduces Modal Auto Endpoints, a self-serve system for deploying production-grade LLM inference. This feature provides users with transparent access to the underlying inference code, metrics, and engine tuning parameters. It leverages Modal's existing AI infrastructure, including high-performance autoscaling and custom container runtime, and introduces Modal Servers for ultra-low-latency routing. The post details the use of advanced speculative decoding techniques like DFlash, highlights engine-level observability with metrics for server performance and inference specifics (TTFT, ITL, speculative decoding acceptance length), and discusses the ongoing development of an agentic system for automated inference endpoint optimization. It also mentions the ability to train custom speculators for specific production data.
  10. Boosting multimodal inference performance by >10% with a single Python dictionary | Modal BlogThis post details an optimization in SGLang's scheduler for multimodal inference. By profiling with `py-spy`, the team identified that expensive book-keeping around shared GPU memory handles during multimodal request processing was a bottleneck. They implemented a Python dictionary cache for these handles, reducing host overhead and improving throughput by 16.2% and reducing mean end-to-end latency by 10.6% on a Qwen2.5-VL-3B-Instruct workload. This optimization was upstreamed to SGLang.
  11. Real-time inference for robots at Physical Intelligence | Modal BlogIntroduced a specialized QUIC-based transport over UDP with automatic NAT traversal for real-time robotic inference. This system establishes a persistent, bidirectional channel between the robot runtime and the Modal GPU container, streaming observations and receiving action outputs over the same channel. It handles automatic NAT traversal using STUN discovery and UDP hole punching, coordinated via Modal Dict for rendezvous. This reduces network overhead to ~10-15ms and avoids potential request-response stalls in control loops.
  12. Runway chooses Modal to power real-time inference for Runway Characters | Modal BlogThis post details how Runway leveraged Modal's infrastructure to power real-time inference for Runway Characters. Key technical contributions include enabling multi-node GPU clusters with RDMA networking via a single line of code, facilitating sustained low latency for expressive video generation across global regions. Modal's platform's ability to distribute inference across multiple GPUs with high-bandwidth communication and route workloads close to users at scale was critical for Runway's success.
  13. How Doppel eliminated ML infrastructure tax with Modal | Modal BlogThis post details how Doppel migrated their ML training and inference workflows to Modal, significantly reducing ML infrastructure tax. For training, it highlights the use of Modal's `map()` function to parallelize K-fold cross-validation and other experiments, shortening feedback loops. For inference, it describes how Modal simplifies deployment by removing the need for manual HTTP service layers, reducing build times through image layer caching and persistent volumes, and automating scaling for traffic spikes. The post contrasts the previous GCP inference stack with the Modal approach, emphasizing faster iteration and reduced operational overhead.
  14. How Reducto improved enterprise-scale document processing latency by 3xThis post details how Reducto migrated their enterprise-scale document processing inference workloads to Modal, achieving a 3x reduction in P90 latency. Key technical contributions include the utilization of GPU memory snapshotting to reduce cold boot times by 83% (from ~70s to ~12s), independent scaling of models by bundling them into single Modal Functions or keeping them separate for full control over autoscaling, and the use of parameterized Functions to create and restrict independent autoscaling compute pools per customer. Additionally, the post highlights the use of granular regional control for latency-sensitive Functions and successful load-testing of the ingestion pipeline to over 1,000 GPUs.
  15. Host overhead is killing your inference efficiencyThis post details how host overhead, characterized by low GPU kernel utilization, negatively impacts AI inference efficiency. It explains that unnecessary synchronization with the CPU and inefficient data transfers between CPU and GPU introduce host overhead. The post provides specific examples of how Modal has reduced host overhead in production inference engines by constructing tensors directly on the GPU to avoid synchronous data transfer and by fusing multiple CUDA kernels into single launches using tools like the Torch compiler and CUDA Graphs. This work aims to minimize microseconds of host overhead per inference pass, contributing to faster and more cost-effective AI inference.
  16. One-second voice-to-voice latency with Modal, Pipecat, and open modelsThis post details the implementation of a low-latency voice AI chatbot achieving sub-second voice-to-voice latency by integrating Modal with the Pipecat framework and open-weight models (Parakeet-tdt-v3 for STT, Qwen3-4B-Instruct-2507 with vLLM for LLM, and KokoroTTS for TTS). Strategies for achieving low latency include using Pipecat's VAD and turn detection with local STT, optimizing LLM inference with vLLM and fine-tuning CUDA graph compilation, and leveraging Modal's independent autoscaling of CPU and GPU functions. The architecture involves receiving audio over WebRTC, transcribing, querying Modal docs via RAG, generating structured LLM responses, and synthesizing speech.
  17. Modal + Datalab: Deploy high-throughput document intelligence in <5 minutesThis post details the deployment of Datalab's Marker and Surya document intelligence models on Modal. It highlights how Modal's platform, specifically its GPU compute capabilities, autoscaling, and caching mechanisms (Modal Volumes), enables high-throughput document processing. The post provides a quickstart guide and explains the underlying technical mechanisms, including environment specification via decorators, GPU usage (e.g., L40s), and the benefits of Modal Volumes for caching model weights. It also quantifies performance gains, showing a 10x throughput increase for Marker on Modal compared to local CPU processing, and discusses how Modal's autoscaling can further improve batch processing throughput.
  18. Transcribe speech 100x faster and 100x cheaper with open modelsThis post details the implementation and optimization of a batch speech transcription service using open-source ASR models (NVIDIA's Parakeet and Canary) on Modal. It highlights achieving over 100x faster or 100x cheaper transcription compared to a proprietary API by leveraging Modal's GPU autoscaling, batch processing capabilities, and optimizing for end-to-end throughput (including cold starts and data transfer). The post contrasts this with a CPU-based approach for saturating a proprietary API and discusses architectural choices for batching and distributing transcription requests.
  19. Dollars per token considered harmfulThis post argues for a shift in thinking from 'dollars per token' to 'dollars per request' when self-hosting LLM inference. It explains how API providers use tokens for pricing but application developers should focus on requests, as that's what users care about and what scales costs. The post details how this request-centric view impacts latency estimation, replica scaling, and cost calculation, advocating for Modal's platform as a solution for achieving lower dollars per request.
  20. Run FLUX.1-dev three times fasterThis post details optimizations applied to the FLUX.1-dev diffusion model for faster image generation. It introduces standard optimizations including Torch compiler integration (`torch.compile` with `max-autotune`), fused QKV projections, and channels-last memory layout, resulting in a 1.5x speedup. Additionally, it implements an approximate caching technique, First Block Caching (adapted from ParaAttention and TeaCache), to achieve a further 2x speedup by skipping steps where image changes are minimal. The post also discusses reducing compile-time for TorchInductor by using a fixed set of configurations and optimizing the `torch.compile` process for faster cold starts.
  21. How Lemon Slice built real-time generative video with Modal and DailyThis post details how Lemon Slice built a real-time generative video product by leveraging Modal's infrastructure for scalable GPU inference and low-latency communication. Key technical contributions include: 1. Using Modal Functions for REST endpoints and video model inference, enabling autoscaling of GPU containers based on request volume. 2. Implementing a real-time pipeline using Pipecat, Deepgram, Grok, and ElevenLabs, with direct container communication via Modal's Tunnels. 3. Co-locating inference containers in specific regions using Modal's region selection feature to minimize latency. 4. Integrating with Daily for streaming the final video and audio back to the user, achieving end-to-end response latencies of 3-6 seconds.
  22. How sync. uses Modal to lipsync 100 hours of video a dayThis post details how sync. uses Modal's parallel execution model to process over 100 hours of video per day for lipsyncing. The workflow involves splitting videos into scenes, running face detection and translation on T4 GPU containers, processing scenes with proprietary LipSync GenAI models on A100 GPUs, and then stitching the dubbed scenes back together. This demonstrates the platform's capability for large-scale, compute-intensive video processing workloads.
  23. 'I paid for the whole GPU, I am going to use the whole GPU': A high-level guide to GPU utilizationThis post introduces a framework for understanding GPU utilization, breaking it down into GPU Allocation Utilization, GPU Kernel Utilization, and Model FLOP/s Utilization. It details strategies for improving each of these, with a focus on neural network inference. For GPU Allocation Utilization, it highlights the importance of fast, automatic allocation and spin-up, and how Modal addresses this with its multi-cloud GPU fleet and custom container stack. For GPU Kernel Utilization, it discusses reducing host overhead and improving GPU throughput through techniques like CUDA Graphs and batching. The post also touches on Model FLOP/s Utilization by discussing the theoretical arithmetic bandwidth of GPUs.
  24. Introducing: L40S GPUs on ModalIntroduced NVIDIA L40S GPUs as a new inference-focused accelerator. The L40S offers 48GB of memory (double the A10) allowing for larger models and inputs, and provides significant speedups for both memory-bound and compute-bound jobs due to its faster Tensor Cores and higher memory bandwidth. Benchmarks show a 20% speedup in a basic load test for chat-style workloads. The post also references a new GPU Glossary for technical vocabulary.
  25. Beat GPT-4o at Python by searching with 100 dumb LLaMAsThis post demonstrates how search techniques, when combined with smaller LLMs (LLaMA 3.1 8B), can achieve performance comparable to or exceeding larger models (GPT-4o) on the HumanEval benchmark. It details the infrastructure used, including Modal's serverless GPUs for LLM inference via vLLM and Modal Sandboxes for secure code execution and evaluation. The results show smooth, predictable scaling of performance with increased search iterations (pass@k), highlighting the industrial relevance of augmenting learning with search. The post also discusses the implications for future AI development, particularly in programming and other domains with precise outcome measurement.
  26. Embedding English Wikipedia in under 15 minutesThis post details how to efficiently generate text embeddings for large datasets like the English Wikipedia using open-source models and Modal's serverless GPU platform. It introduces a workflow for downloading and storing datasets in Modal Volumes, setting up Hugging Face's Text Embedding Inference server on Modal GPUs (A10G), and processing data in batches for optimized embedding generation. The contribution focuses on practical implementation for large-scale embedding tasks, reducing generation time from hours to minutes and demonstrating cost-effectiveness.