Continuous vs dynamic batching for AI inference

This post introduces and contrasts four methods of batching inference requests: no batching, static batching, dynamic batching, and continuous batching. It details the implementation and use cases for each, emphasizing continuous batching for LLMs due to their token-by-token generation and dynamic batching for models with fixed inference times. It also mentions existing opportunities for concurrency in end-to-end systems and references specific inference servers like TGI and VLLM that support continuous batching.