How we run GPT OSS 120B at 500+ tokens per second on NVIDIA GPUs

This post details the engineering effort to achieve state-of-the-art latency and throughput for GPT OSS 120B on NVIDIA GPUs. It covers the process of testing and benchmarking across inference frameworks (TensorRT-LLM, vLLM, SGLang), ensuring compatibility with Hopper and Blackwell GPU architectures, integrating with NVIDIA Dynamo, and implementing performance optimizations such as KV cache-aware routing and speculative decoding with Eagle. The post also discusses the trade-offs between Tensor Parallelism and Expert Parallelism for mixture-of-experts models, and the contribution of bug fixes back to open-source projects.