TechPrufer
Get started
Modal
Modal logoModal · Feature trail

Dynamic Batching

4 posts · Sep 2024 – May 2025

What this trail covers

Modal's platform supports massive job processing with a new durable queue system and a `.spawn_map` interface, capable of handling 1 million inputs and guaranteeing execution for up to 7 days. This significantly enhances the platform's ability to manage large-scale batch workloads, building upon existing dynamic batching capabilities by providing a more robust and scalable solution for tasks like data preprocessing, embedding generation, and training data preparation. FunctionCall results now pe

All Modal trails
  1. Introducing Modal Batch: Process 1 million jobs with 1 line of codeIntroduces Modal Batch, a new interface backed by a durable queue system for massive job processing. Enhancements include a 500x increase in input capacity (1 million inputs), guaranteed execution for up to 7 days, and a native `.spawn_map` handler for batch input processing. This builds upon existing dynamic batching capabilities by providing a more robust and scalable solution for tasks like data preprocessing, embedding generation, and training data preparation.
  2. Product updates: Running batch jobs with 1M inputs, ephemeral apps, and a new TensorRT-LLM exampleIncreased the maximum number of inputs per Modal Function to 1 million (from 2k). Raised the `.spawn()` rate limit for faster input submission. Extended `FunctionCall` result persistence to 7 days.
  3. Boost your throughput with dynamic batchingThis post details the implementation and benefits of adding native dynamic batching support to Modal. It explains the core concept of batching, why dynamic batching is superior to traditional batching for sporadic request patterns, and demonstrates its application to OpenAI's Whisper model. The contribution involves adding a `@modal.batched()` decorator to inference functions, which requires the function to accept and return lists of samples/results. This change, with a `max_batch_size` of 64 and `wait_ms` of 1000, achieved a 2.8x increase in throughput for Whisper inference on an A10G GPU, leading to significant cost savings.
  4. Product updates: Rollbacks, batching, sandbox tunnels & moreIntroduces the `@batched` decorator which accumulates requests into dynamically-sized batches, allowing users to trade off latency for throughput and GPU utilization. Also adds the ability to forward TCP ports from `Sandbox` objects to the internet and `exec` into existing `Sandbox` objects, with a stated interest in improving Sandboxes for agentic use cases.