Model Quantization: Turn FP8 Checkpoints into High-Performance Inference Engines with NVIDIA TensorRT | NVIDIA Technical Blog

What this post added

This post details the practical steps for converting an FP8-quantized CLIP model checkpoint into a TensorRT inference engine. It covers exporting the model to ONNX format with QuantizeLinear/DequantizeLinear (Q/DQ) nodes, explaining how TensorRT fuses these nodes for direct execution on FP8 Tensor Cores. The post also includes benchmarking results using `trtexec` on an NVIDIA RTX 6000 Ada GPU, demonstrating a 1.39x-1.45x speedup in inference latency for FP8 engines compared to FP16, attributed to optimized FP8 matmul and attention kernels.