Understanding performance benchmarks for LLM inference

This post provides a detailed guide on understanding and creating nuanced performance benchmarks for LLM inference. It breaks down key factors to consider, including hardware, streaming, quantizing, input/output size, batch size, percentile reporting, and network speed. It elaborates on measuring latency (time to first token, total generation time, cold starts) and throughput (requests per second, tokens per second), emphasizing the importance of output tokens per second for real-time applications. The post also discusses cost measurement (cost per minute, cost per thousand tokens) and provides an example of a high-quality benchmark definition.