Does Quantization Speed Up Inference? A Practical Guide

This post details model quantization for AI inference, explaining its mechanism for speeding up inference by reducing data movement and computation. It covers various quantization formats (FP32, FP16, INT8, INT4), their impact on accuracy and performance, and scenarios where it's most effective. The post contrasts quantization with other optimization techniques like batching and caching, emphasizing that quantization pays off most when the model is the bottleneck. It also discusses the importance of hardware support and the potential for slowdowns due to software conversion overhead when native support is lacking. The post highlights FP8 as a promising format on newer GPUs and INT8 as a safer default for older hardware. Accuracy trade-offs are discussed, with FP8 being nearly lossless and INT4 showing larger drops on specific tasks. Quantization-aware training (QAT) is mentioned as a method to recover lost accuracy.