Introduction to quantizing ML models

This post introduces the concept of ML model quantization, explaining how it reduces model size and improves inference speed by mapping model weights to lower precision formats (e.g., FP16 to INT8/INT4). It details the benefits, such as reduced memory bandwidth and increased compute, and discusses the trade-offs, particularly the potential for reduced output quality. The post also outlines practical aspects like using TensorRT-LLM and weights-only quantization, and provides guidance on when to apply quantization for production inference.