Benchmarking LLM Inference at Scale with AIPerf | NVIDIA Technical Blog

This post introduces AIPerf, a new multiprocess benchmarking tool that replaces GenAI-Perf. AIPerf addresses client-side bottlenecks by using a multiprocess architecture with separate worker processes for load generation and record-processor services for results handling, coordinated via ZMQ. It supports over 15 endpoint types and various dataset formats for realistic workload testing, along with configurable arrival patterns (constant, Poisson, gamma) and burstiness. The tool reports core metrics like TTFT, ITL, request latency, and output token throughput with percentile breakdowns and GPU telemetry (when DCGM or pynvml is available). The post provides a walkthrough of running a synthetic ISL/OSL benchmark on vLLM with Qwen3-0.6B, detailing installation and command-line usage.