How we made the fastest GPT-OSS on NVIDIA GPUs 60% faster

This post details the implementation of EAGLE-3 speculative decoding to improve GPT-OSS 120B inference speed by 60% on NVIDIA B200 GPUs, achieving over 650 TPS. It explains the mechanics of speculative decoding, the benefits of EAGLE-3's architecture (size, accuracy via hidden state utilization), and the integration challenges with TensorRT-LLM. The post also discusses the trade-offs between NVIDIA GPUs and custom hardware, emphasizing Baseten's ability to deliver performance, flexibility, and scale on generally available hardware with full context windows. It highlights independent benchmarking results from Artificial Analysis and OpenRouter, showcasing real-world performance.