Boost your throughput with dynamic batching

This post details the implementation and benefits of adding native dynamic batching support to Modal. It explains the core concept of batching, why dynamic batching is superior to traditional batching for sporadic request patterns, and demonstrates its application to OpenAI's Whisper model. The contribution involves adding a `@modal.batched()` decorator to inference functions, which requires the function to accept and return lists of samples/results. This change, with a `max_batch_size` of 64 and `wait_ms` of 1000, achieved a 2.8x increase in throughput for Whisper inference on an A10G GPU, leading to significant cost savings.