Modal Feature Trails

Feature trailActive

Inference Performance Optimization

Modal's platform enables significant performance improvements for AI inference through various optimizations. This includes reducing host overhead in schedulers, optimizing multimodal inference by caching shared GPU memory handles, enhancing speculative decoding for LLMs, enabling serverless GPU inference with rapid spin-up times, and supporting real-time generative video by co-locating inference containers and using direct communication via Tunnels. Recent work has focused on optimizing diffusion models and scaling to trillions of tokens for System One models like Jev, handling bursty, high-concurrency workloads with ultra-low-latency inference without managing infrastructure.

20 posts · 2024–2026

Timeline

20242026

Feature trailActive

Sandboxed Code Execution

Modal's platform supports scaling to millions of concurrent sandboxes, with tens of thousands of sandboxes created per second. This is achieved by rebuilding the core sandbox platform from the ground up, removing central bottlenecks from the control plane, and optimizing container scheduling and startup. The new system utilizes a fleet of horizontally scaled scheduling servers that directly contact workers via RPC, bypassing central datastores in the critical path for sandbox creation. VM Sandboxes can now run alongside Sidecars, which provide a low-latency, isolated security boundary for trusted code, enabling scenarios like custom egress proxies and separating agent harnesses from tool calls. Sidecars share CPU and memory resources with the main Sandbox and can be created dynamically at runtime.

29 posts · 2024–2026

Timeline

20242026

Feature trailActive

AI Research Infrastructure

Modal's platform provides scalable, reproducible, and isolated infrastructure for executing AI research workloads, including GPU compute, consistent benchmarking environments, and sandboxed execution. This enables researchers to accelerate AI development, particularly in areas like automated kernel generation and self-improving AI systems. The platform now offers enhanced support for Reinforcement Learning (RL) post-training of LLMs, addressing infrastructure challenges in training, rollouts, and deployment. Substack has migrated both training and deployment of their ML models from AWS SageMaker to Modal, significantly improving developer experience, collaboration, and iteration speed. They implemented a full fine-tune → validation → deployment workflow leveraging Modal's native storage primitives and key-value store for tracking production model versions. This has enabled faster development and deployment of ML workflows, including a new recommendation model in days.

7 posts · 2024–2026

Timeline

20242026

Feature trailActive

Serverless GPU Inference

Modal's platform enables truly serverless GPU inference by optimizing replica spin-up times to tens of seconds. This is achieved through a combination of cloud buffers for pre-warmed GPUs, a custom lazy-loading filesystem, and advanced checkpoint/restore mechanisms for both CPU and CUDA contexts. Recent work has introduced GPU memory snapshotting, which can reduce median cold start times for certain workloads (like LLM inference with vLLM) by almost 10x, from minutes to seconds, by saving and re-using model states. This post demonstrates how to leverage this infrastructure with smaller models and search to achieve frontier-level performance on code generation tasks, matching or exceeding GPT-4o's pass@1 performance on HumanEval with 100 LLaMA 3.1 8B generations.

11 posts · 2024–2026

Timeline

20242026

Feature trailActive

Speculative Decoding for LLM Inference

Modal's platform deeply integrates speculative decoding for LLM inference, enabling significant speedups by using a draft model to generate token sequences in parallel. This feature allows for lossless acceleration of the autoregressive decoding phase, turning serial work into parallel work. The platform supports training custom speculators tailored to application-specific data, leading to substantial improvements in acceptance lengths and overall inference throughput. Recent work has focused on optimizing diffusi

8 posts · 2025–2026

Timeline

20252026

Feature trailActive

Region Selection

Modal's platform allows users to select cloud regions for Function execution, offering granular control over deployment locations to meet regulatory requirements, reduce egress fees, and minimize latency by co-locating compute with data or users. This feature supports specifying regions at various levels of detail, from broad continents to specific data centers. Region selection is now available on all plans. Recent work has focused on validating the benefits of a distributed control plane to further optimize region selection and resource allocation.

5 posts · 2024–2026

Timeline

20242026

Feature trailActive

Developer Experience for AI Agents

Modal's platform is evolving to enhance developer experience not just for human developers but also for AI coding agents. This involves applying core principles like fast feedback loops, programmatic access, actionable error messages, illustrative examples, unified infrastructure and logic, and consistent naming conventions. The goal is to accelerate the iterative process for both humans and agents, making the development lifecycle more efficient and less error-prone, particularly in AI development. Recent work has focused on applying these principles to agentic workflows and tooling.

1 post · 2025–2026

Timeline

20252026

Feature trailActive

Virtual Computer Abstraction

Modal's platform is presented as a virtual computer, abstracting and multiplexing compute resources across multiple cloud providers. This layer of virtualization aggregates, isolates, and multiplexes demand onto hardware at a grander scale than individual clouds, enabling higher resource efficiency for engineering teams by providing a more productive abstraction over raw cloud infrastructure. This post elaborates on the analogy of Modal as a computer, detailing how programs run, how they are stored, and how they are executed. Chai Discovery leverages this abstraction to seamlessly scale complex, bursty bio workloads, including diverse ML models and large biological datasets, from research through production. They utilize Modal Volumes for high-throughput, shared data access, eliminating repetitive data setup and ensuring consistent, reproducible execution environments. Dynamic GPU scaling and workload elasticity allow them to handle variable demand without manual orchestration or capacity planning. The platform powers everything from large-scale model training experiments to molecular design inference pipelines, enabling fast feedback loops and moving research ideas into production with minimal friction.

2 posts · 2026

Timeline

Feature trailActive

WebSockets

Modal functions now support the WebSocket protocol, enabling real-time, bidirectional data transfer between clients and servers. This feature allows for persistent connections, reducing latency and facilitating real-time updates for applications. It integrates seamlessly with existing Modal functions, supporting libraries like FastAPI and automatically scaling based on input volume. Use cases include real-time streaming responses for audio and text generation, status updates on long-running task. This post demonstrates building a low-latency voice AI chatbot using Pipecat and open models, leveraging Modal's ability to autoscale independent GPU and CPU services for STT, LLM, and TTS inference, and integrating with WebSockets for real-time communication.

3 posts · 2024–2026

Timeline

20242026

Feature trailActive

Flash Attention Kernel Analysis

Modal's platform provides deep integration with FlashAttention kernels, enabling significant performance optimizations for LLM inference. Recent contributions focus on adapting FlashAttention-4 for decode-heavy workloads by adjusting parallelism strategies (e.g., switching to KV parallelism) and supporting irregular global memory accesses via `cp.async` and Tensor Memory Accelerator (TMA) optimizations. This includes adding support for FP8 inputs, enabling arbitrary KV page sizes, and optimizing for Nvidia's Blackwell architecture. The platform also details the asynchronous pipeline architecture of FA4, including warp specialization and producer-consumer models for efficient execution.

3 posts · 2025–2026

Timeline

20252026

Feature trailActive

Role-Based Access Control (RBAC)

Modal now offers Role-Based Access Control (RBAC) for Team and Enterprise plans, building on the concept of Environments as secure access boundaries. This system allows for granular control over human and agent access to Modal resources, enabling the creation of Restricted Environments where only explicitly granted users or service users can deploy or manage applications. RBAC integrates with Audit Logs for comprehensive tracking and is the foundation for future agent-centric development tools. The new architecture supports spending and time-based limits per agent, programmatic access to permission primitives, and resource-based permissions, with Restricted Environments becoming the default.

1 post · 2026

Timeline

Feature trailActive

GPU Health and Reliability System

Modal runs a globally distributed, autoscaling GPU worker pool by sourcing compute from all cloud giants: AWS, GCP, Azure, OCI. We've scaled the worker pool to well over 20,000 concurrent GPUs, and launched over four million cloud instances in the last couple years. At this scale, you see almost every GPU reliability problem there is. This post shares our GPU reliability system as both a demonstration of our commitment to Modal customers and as a guide for fellow travelers renting hyperscaler or neocloud cards. This post starts with cloud instance type testing and selection. Perhaps surprisingly, there are significant performance and reliability differences between the cloud hyperscalers. We then discuss machine image preparation and instance boot checks. Next we cover the passive and active GPU healthchecking performed throughout the life of each instance. Finally we discuss observability and support, which become crucial when a GPU reliability issue slips by our automated healthchecking systems. We've chosen not to refer to cloud providers directly, but instead give them anonymized A, B, C, D identifiers. If you want know who's who, track the clues or buy us a beer sometime.

1 post · 2025–2026

Timeline

20252026

Feature trailActive

ETL Platform

Modal's platform supports eStargz compression for faster image builds from external registries like ECR and Docker Hub, and introduces OpenID Connect (OIDC) for secure authentication against external resources, reducing the need for long-lived credentials. The platform also enhances memory snapshotting for faster cold starts and expands capabilities for batch processing and job queues with a higher input limit. This post highlights the use of Modal for large-scale spatio-temporal data processing and demonstrates cost-effective ETL pipelines by comparing custom code on Modal against vendor solutions like Fivetran, showcasing significant cost savings for large data volumes and flexible custom code execution.

4 posts · 2024–2026

Timeline

20242026

Feature trailCooling

Generative AI Evals and Compute Scaling

Modal's platform supports advanced techniques for engineering generative AI applications, including robust evaluation frameworks ('evals') and dynamic inference-time compute scaling. This enables developers to systematically improve the quality and performance of AI-driven systems, such as generating aesthetically pleasing and functional QR codes. The platform facilitates the development lifecycle from initial demos to production-ready systems by providing tools for objective measurement, and has been used to achieve high QR code scan rates and aesthetic quality.

1 post · 2025

Timeline

20252026

Feature trailCooling

Client API Stability and Ergonomics

Modal 1.0 marks a significant milestone in the maturity and stability of the Modal platform's client API. Key design principles for this release include avoiding unpredictable 'magic' behaviors by making features like package automounting explicit, separating core functionality from additional concepts through distinct decorators (e.g., `@modal.concurrent`), and consolidating related functionalities into canonical paths (e.g., managing local files within `modal.Image`). This release aims to provide a more predictable and easier-to-debug developer experience, with a more batched and predictable client release cycle.

2 posts · 2025

Timeline

20252026

Feature trailCooling

Dynamic Batching

Modal's platform supports massive job processing with a new durable queue system and a `.spawn_map` interface, capable of handling 1 million inputs and guaranteeing execution for up to 7 days. This significantly enhances the platform's ability to manage large-scale batch workloads, building upon existing dynamic batching capabilities by providing a more robust and scalable solution for tasks like data preprocessing, embedding generation, and training data preparation. FunctionCall results now pe

4 posts · 2024–2025

Timeline

20242026

Feature trailCooling

Serverless Key-Value Store (Dict)

Modal's Dict primitive provides a serverless TTL'ed key-value store accessible from any container within a workspace. Recent enhancements include unlimited storage, a revised expiry policy (7 days since last read or write), data durability, and a locking primitive via the `.put()` method's `skip_if_exists` flag. These improvements enable advanced use cases like LRU-like caching with extended TTLs for hot entries and distributed locking for 'exactly once' semantics.

1 post · 2025

Timeline

20252026

Feature trailCooling

Resource Solver

Modal's platform utilizes a sophisticated resource solver, powered by linear programming (LP) and the GLOP solver, to dynamically manage cloud compute resources. This system optimizes instance procurement based on real-time demand, pricing, and availability across multiple cloud providers. It ensures customers receive scalable compute at predictable and favorable prices, while also identifying arbitrage opportunities to reduce operational costs. The solver balances the dual mandates of rapid scaling and cost-optimization, employing heuristics to prune instance types and ensure fast solve times. Observed scaling limits from cloud providers are fed back into the solver for subsequent runs. Background workers request instances based on the solver's output, and report observed limits back.

1 post · 2025

Timeline

20252026

Feature trailCooling

Fine-tuning Generative Models

Modal's platform supports fine-tuning generative models on custom datasets, enabling users to prepare datasets, set up containerized environments with necessary dependencies, utilize cloud storage for model weights, and configure training hyperparameters. The process includes training scripts, hyperparameter configuration, and serving the fine-tuned model via a web UI. Ramp has utilized Modal to accelerate the development of their text-to-structured-JSON model for receipt management, driving down operational costs and improving accuracy. This post details an experiment to beat proprietary embedding models with a quick fine-tune, demonstrating that with just a handful of examples, a fine-tuned open-source embedding model can provide greater accuracy at a lower price than proprietary models like OpenAI's `text-embedding-3-small`.

7 posts · 2023–2024

Timeline

20232026

Feature trailCooling

Static IP Proxies

Modal's platform supports static IP egress for serverless containers, enabling secure connections to private resources. This is achieved through a Go-based VPN proxy (_vprox_) utilizing WireGuard for Layer 3 tunnels, handling dynamic IP allocation and policy-based routing. The system ensures all outbound traffic originates from a static IP, addressing requirements for consistent egress IPs.

2 posts · 2024

Timeline

20242026

Feature trailCooling

Cloud Function Platform

Modal's cloud function platform supports automated CI workflows on GPUs, enabling parallelized testing and faster iteration for AI development. This integration allows GitHub Actions to trigger Modal Functions with custom GPU requirements, significantly reducing test execution time and eliminating the need for self-hosted runners. The platform's usage-based billing and cached image builds contribute to cost-effectiveness.

9 posts · 2024

Timeline

20242026

Feature trailQuiet since 2024

Hybrid Search with Embeddings

Modal's platform supports building hybrid search applications by integrating with data platforms like MongoDB Atlas and foundation models for generating embeddings. This enables complex search queries combining text, geolocation, timestamps, and vector embeddings derived from satellite imagery, demonstrating a new capability for data-driven AI applications. This post details a specific implementation using Sentinel satellite imagery, Clay v1 foundation model, and MongoDB Atlas for hybrid search over California.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Runtime Abuse Detection

Modal has implemented a syscall-based program analysis component called `seccheck` within its serverless container runtime. This system leverages gVisor's syscall interception capabilities to monitor and detect cryptomining activities by analyzing syscall patterns. The `seccheck` component identifies suspicious syscall sequences, such as specific file access patterns or network connections associated with mining pools, and can automatically disable offending Functions, ban users, and prevent further abuse. This builds on earlier metadata-based detection systems.

1 post · 2024

Timeline

20242026