Dynamic Batching
Boost your throughput with dynamic batching

Boost your throughput with dynamic batching

9/16/2024

What this post added

Introduced native dynamic batching support to Modal. This involves adding a `@modal.batched()` decorator to inference functions, which takes `max_batch_size` and `wait_ms` parameters. The inference function signature was updated to accept and return lists of samples and results, respectively. This feature was integrated into a Whisper transcription example to demonstrate its impact on throughput and cost.

Read the original post ↗