Modal Feature Trails

Modal now allows users to select the cloud region where their Functions execute, offering granular control over deployment locations. This feature supports specifying regions at various levels of detail, from broad continents to specific data centers, to meet regulatory requirements, reduce egress fees, and minimize latency by co-locating compute with data or users. The implementation involves a simple code-based configuration, enabling users to direct their workloads to desired geographical zones. This post details the rollout of a new, geographically distributed I/O plane that further reduces latency by routing Function inputs and outputs through regional I/O servers, complementing the existing region selection for container execution. End-to-end Function Call latencies have dropped by approximately 80ms at p50 due to this new I/O plane.

Modal's platform now deeply integrates speculative decoding for LLM inference, enabling significant speedups by using a draft model to generate token sequences in parallel. This feature allows for lossless acceleration of the autoregressive decoding phase, turning serial work into parallel work. The platform supports training custom speculators tailored to application-specific data, leading to substantial improvements in acceptance lengths and overall inference throughput. This is achieved through integration with models like Kimi K3, where custom DFlash speculators tuned to the model's architecture are employed, resulting in significant interactivity and throughput improvements.

Modal's platform now supports scaling to millions of concurrent sandboxes, with tens of thousands of sandboxes created per second. This is achieved by rebuilding the core sandbox platform from the ground up, removing central bottlenecks from the control plane, and optimizing container scheduling and startup. The new system utilizes a fleet of horizontally scaled scheduling servers that directly contact workers via RPC, bypassing central datastores in the critical path for sandbox creation. Recent updates include enhanced networking capabilities for sandboxes, allowing secure HTTP and WebSocket connections via Sandbox Connect Tokens, and the integration of Modal Sandboxes into various AI agent and coding platform demos, demonstrating their ability to securely run code at scale and connect to UIs through Modal Tunnels.

Modal is presented as a virtual computer, abstracting and multiplexing compute resources across multiple cloud providers. This layer of virtualization aggregates, isolates, and multiplexes demand onto hardware at a grander scale than individual clouds, enabling higher resource efficiency for engineering teams by providing a more productive abstraction over raw cloud infrastructure.

Modal's platform enables truly serverless GPU inference by optimizing replica spin-up times to tens of seconds. This is achieved through a combination of cloud buffers for pre-warmed GPUs, a custom lazy-loading filesystem, and advanced checkpoint/restore mechanisms for both CPU and CUDA contexts. Recent work has introduced GPU memory snapshotting, which can reduce median cold start times for certain workloads (like LLM inference with vLLM) by almost 10x, from minutes to seconds, by saving and re-using GPU memory. This post further elaborates on the economic advantages of serverless GPUs by introducing a cost model that compares serverless GPU pricing against reserved instances, highlighting that serverless GPUs are often cheaper when the peak-to-average demand ratio exceeds the reservation discount rate. It also discusses the assumptions and limitations of this cost model, including perfect demand prediction and instant allocation, and touches upon the developer productivity benefits of serverless GPU usage.

Modal's platform provides scalable, reproducible, and isolated infrastructure for executing AI research workloads, including GPU compute, consistent benchmarking environments, and sandboxed execution. This enables researchers to accelerate AI development, particularly in areas like automated kernel generation and self-improving AI systems. The platform now offers enhanced support for Reinforcement Learning (RL) post-training of LLMs, addressing infrastructure challenges in training, rollouts, and multi-node training clusters. It also supports the latest NVIDIA B200 and H200 GPUs for LLM inference and fine-tuning, and has released version 1.0 of its client API for enhanced stability and ergonomics.

Modal functions now support the WebSocket protocol, enabling real-time, bidirectional data transfer between clients and servers. This feature allows for persistent connections, reducing latency and facilitating real-time updates for applications. It integrates seamlessly with existing Modal functions, supporting libraries like FastAPI and automatically scaling based on input volume. Use cases include real-time streaming responses for audio and text generation, status updates on long-running task. Modal now also offers ultra-low-latency Servers for HTTP, WebSocket, and gRPC traffic, providing a regionalized, autoscaling pool of server replicas behind Modal's routing layer. This new Server capability reduces p50 latency from 39ms to 6ms by minimizing overhead and pushing queueing/retries to the application layer, making it ideal for applications like LLM inference where every millisecond counts. The Server architecture involves an L4 NLB, an Envoy edge proxy for TLS termination and HTTP/2 normalization, and an in-house proxy `fprs` built with Pingora for domain association and load balancing across compute plane replicas.

Modal's platform enables significant performance improvements for AI inference through various optimizations. This includes reducing host overhead in schedulers, optimizing multimodal inference by caching shared GPU memory handles, enhancing speculative decoding for LLMs, enabling serverless GPU inference with rapid spin-up times, and supporting real-time generative video by co-locating inference containers and using direct communication via Tunnels. Recent work has focused on optimizing diffusion models and reducing ML infrastructure tax for training and inference workloads by simplifying deployment, enabling parallel experimentation, and automating scaling.

Modal's platform provides deep integration with FlashAttention kernels, enabling significant performance optimizations for LLM inference. Recent contributions focus on adapting FlashAttention-4 for decode-heavy workloads by adjusting parallelism strategies (e.g., switching to KV parallelism) and supporting irregular global memory accesses via `cp.async` and Tensor Memory Accelerator (TMA) optimizations. This includes adding support for FP8 inputs, enabling arbitrary KV page sizes, and optimizing for LLM inference, including support for models like GLM-5. The platform also offers a free endpoint for GLM-5 and sample deployment code.

Modal now offers Role-Based Access Control (RBAC) for Team and Enterprise plans, building on the concept of Environments as secure access boundaries. This system allows for granular control over human and agent access to Modal resources, enabling the creation of Restricted Environments where only explicitly granted users or service users can deploy or manage applications. RBAC integrates with Audit Logs for comprehensive tracking and is the foundation for future agent-centric development tools.

Modal's platform now supports eStargz compression for faster image builds from external registries like ECR and Docker Hub, and introduces OpenID Connect (OIDC) for secure authentication against external resources, reducing the need for long-lived credentials. The platform also enhances memory snapshotting for faster cold starts and expands capabilities for batch processing and job queues with a higher input limit. This post highlights the use of Modal for large-scale spatio-temporal data processing on GPUs, demonstrating efficient batch processing and cost reduction.

Modal has developed a comprehensive system for maintaining the health and reliability of its globally distributed GPU worker pool, encompassing over 20,000 GPUs. This system includes rigorous instance type testing and selection, robust machine image preparation and automated testing, lightweight instance boot checks, and continuous lifetime management through passive and active healthchecking. Observability is provided via detailed GPU metrics for customers and internal dashboards. The system aims to proactively identify and mitigate GPU hardware issues across multiple cloud providers.

Modal's platform is evolving to enhance developer experience not just for human developers but also for AI coding agents. This involves applying core principles like fast feedback loops, programmatic access, actionable error messages, illustrative examples, unified infrastructure and logic, and consistent naming conventions. The goal is to accelerate the iterative process for both humans and agents, making the development lifecycle more efficient and less error-prone, particularly in AI development workflows.

Modal's platform now supports advanced techniques for engineering generative AI applications, including robust evaluation frameworks ('evals') and dynamic inference-time compute scaling. This enables developers to systematically improve the quality and performance of AI-driven systems, such as generating aesthetically pleasing and functional QR codes. The platform facilitates the development lifecycle from initial demos to production-ready systems by providing tools for objective measurement, alignment with human judgment, and scalable execution.

Modal 1.0 marks a significant milestone in the maturity and stability of the Modal platform's client API. Key design principles for this release include avoiding unpredictable 'magic' behaviors by making features like package automounting explicit, separating core functionality from additional concepts through distinct decorators (e.g., `@modal.concurrent`), and consolidating related functionalities into canonical paths (e.g., managing local files within `modal.Image`). This release aims to prov. The platform now offers official SDKs for JavaScript and Go, enabling developers to call deployed Modal Functions and create isolated sandboxes from server-side JavaScript or Go code. This expands Modal's accessibility beyond Python, allowing integration with Node backends, Next.js server handlers, and Go microservices. The SDKs support calling functions with JSON-like or bytes data types and creating sandboxes for executing code, running untrusted scripts, checking out Git repos, and running one-off jobs. While deployment of Modal Functions still requires Python, the SDKs provide a lightweight, consistent, and easy-to-use interface for interacting with Modal from other languages.

Modal Batch introduces a new durable queue system and a `.spawn_map` interface for massive job processing, capable of handling 1 million inputs and guaranteeing execution for up to 7 days. This significantly enhances the platform's ability to manage large-scale batch workloads, building upon existing dynamic batching capabilities by providing a more robust and scalable solution for tasks like data preprocessing, embedding generation, and training data preparation. FunctionCall results now persist for 7 days.

Modal's Dict primitive, a serverless TTL'ed key-value store accessible from any container within a workspace, has been significantly enhanced. New Dicts now offer unlimited storage, a revised expiry policy (7 days since last read or write), and introduce a locking primitive via the `.put()` method's `skip_if_exists` flag. Data durability has also been added. These improvements enable advanced use cases like LRU-like caching with extended TTLs for hot entries and distributed locking for 'exactly once' semantics in redundant operations. The post demonstrates building a request cache that leverages these new features to de-duplicate and cache expensive function call results, reducing backend load and improving customer experience.

Modal's platform now utilizes a sophisticated resource solver, powered by linear programming (LP) and the GLOP solver, to dynamically manage cloud compute resources. This system optimizes instance procurement based on real-time demand, pricing, and availability across multiple cloud providers. It ensures customers receive scalable compute at predictable and favorable prices, while also identifying arbitrage opportunities to reduce operational costs. The solver balances the dual mandates of rapid scaling and cost-efficiency, incorporating heuristics and feedback loops to maintain predictable solve times and adapt to observed cloud provider capacity limits.

Modal now supports fine-tuning generative models on custom datasets, enabling users to prepare datasets, set up containerized environments with necessary dependencies, utilize cloud storage for model weights, and configure training hyperparameters. The process includes training scripts, hyperparameter configuration, and serving the fine-tuned model via a web UI. Ramp has utilized Modal to accelerate the development of their text-to-structured-JSON model for receipt management, driving down receipt processing costs. This post details the process of fine-tuning LLMs, including cost benefits, common use cases, and steps for preparing datasets and training models, with a focus on using Modal for infrastructure.

Modal now offers Static IP Proxies, enabling serverless containers to egress traffic through consistent, globally distributed IPv4 addresses. This is achieved by deploying a Go-based VPN proxy (_vprox_) utilizing WireGuard to establish Layer 3 tunnels between containers and proxy nodes. The system handles dynamic IP allocation, peer management, and policy-based routing on worker machines to ensure all outbound traffic from containers originates from a static IP. This addresses scenarios requiring stable egress IPs for database connections and other private resource access. The latest update allows for workspace-exclusive static IPs.

Modal's cloud function platform now supports automated CI workflows on GPUs, enabling parallelized testing and faster iteration for AI development. This integration allows GitHub Actions to trigger Modal Functions with custom GPU requirements, significantly reducing test execution time and eliminating the need for self-hosted runners. The platform's usage-based billing and cached image builds contribute to cost-effectiveness.

Modal now supports building hybrid search applications by integrating with data platforms like MongoDB Atlas and foundation models for generating embeddings. This enables complex search queries combining text, geolocation, timestamps, and vector embeddings derived from satellite imagery, demonstrating a new capability for data-driven AI applications.

Modal has implemented a syscall-based program analysis component called `seccheck` within its serverless container runtime. This system leverages gVisor's syscall interception capabilities to monitor and detect cryptomining activities by analyzing syscall patterns. The `seccheck` component identifies suspicious syscall sequences, such as specific file access patterns or network connections associated with mining pools, and can automatically disable offending Functions, ban users, and prevent further abuse. This enhances Modal's defenses against resource abuse and ensures GPU availability for legitimate users.