How to run LLM performance benchmarks (and why you should)

This post details the process of replicating the InferenceMAX LLM performance benchmark on Baseten using TensorRT-LLM. Key contributions include: provisioning 4xB200 GPUs, configuring `config.yaml` to match the InferenceMAX TensorRT-LLM image, adapting benchmark scripts to use the `/completions` endpoint and `ignore_EOS` flag, and trussing the deployment. It introduces server-side benchmarking against `localhost:8000/v1` to isolate inference performance from network variability, contrasting it with client-side benchmarking for user experience. The post also emphasizes the importance of developer experience constructs like caching (using `model_cache` for GPT-OSS-120B), server-side benchmarking, and scripting for iterative development, and discusses dataset selection strategies (production, public, synthetic data).