
9/16/2024
What this post added
Introduced native dynamic batching support to Modal. This involves adding a `@modal.batched()` decorator to inference functions, which takes `max_batch_size` and `wait_ms` parameters. The inference function signature was updated to accept and return lists of samples and results, respectively. This feature was integrated into a Whisper transcription example to demonstrate its impact on throughput and cost.