Model Quantization: Turn FP8 Checkpoints into High-Performance Inference Engines with NVIDIA TensorRT | NVIDIA Technical Blog

This post details the practical steps for converting FP8-quantized CLIP checkpoints into high-performance TensorRT inference engines. It covers exporting the ModelOpt checkpoint to ONNX format, highlighting the resulting ~34-50% reduction in ONNX file size due to folding Q-DQ nodes. It also describes profiling the FP8 TensorRT engine using `trtexec` and Nsight Deep Learning Designer, demonstrating a 1.39x-1.45x speedup in inference latency for CLIP encoders compared to FP16, attributed to optimized FP8 matmul and attention kernel execution.