33% faster LLM inference with FP8 quantization

This post details the successful implementation and validation of FP8 quantization for LLM inference, specifically for Mistral 7B on NVIDIA H100 GPUs using TensorRT-LLM. It quantifies performance improvements (33% speed increase, 8.5% TTFT decrease, 24% cost reduction) and demonstrates rigorous quality assurance through perplexity benchmarks and anecdotal evaluations, showing FP8's viability for production workloads without significant degradation. The post also discusses previous INT8 quantization challenges and the advantages of FP8's dynamic range.