Dynamic Batching

5/22/2025

Introduced Modal Batch, a new interface backed by a new durable queue system designed for large-scale, fault-tolerant job processing. Key improvements include an overhauled queue system supporting 1 million inputs (up from 2000), guaranteed input execution for up to 7 days (up from 1 day), and a native `.spawn_map` handler for batch job initiation. This builds upon the existing dynamic batching feature by providing a more robust and scalable solution for massive jobs.

4/17/2025

Increased the maximum number of inputs per Modal Function to 1 million, raised the `.spawn()` rate limit, and extended the persistence of `FunctionCall` results to 7 days.

9/16/2024

Introduced native dynamic batching support to Modal. This involves adding a `@modal.batched()` decorator to inference functions, which takes `max_batch_size` and `wait_ms` parameters. The inference function signature was updated to accept and return lists of samples and results, respectively. This feature was integrated into a Whisper transcription example to demonstrate its impact on throughput and cost.

9/6/2024

Introduced the `@batched` decorator for dynamic batching, which accumulates requests into dynamically-sized batches to trade off latency for throughput and GPU utilization.