Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

This post details critical flaws in existing LLM performance benchmark harnesses, including issues with QPS reporting, client-side latency inflation, and data sampling discrepancies. It introduces 'Inference Perf,' a CNCF project designed as a multiprocess load generator with client and server telemetry for accurate measurement. It also highlights the need for a workload catalog to define agentic generation, tree of thought, and batch summarization in a standardized way, and mentions 'Prism' for visualizing optimizations.