FP8: Efficient model inference with 8-bit floating point numbers

This post introduces FP8 as an efficient data format for model inference, offering an alternative to INT8 for quantizing LLMs. FP8's higher dynamic range allows for more efficient quantization of activations and KV cache, leading to faster inference and reduced operational costs, especially for smaller models, with less degradation of output quality. FP8 is supported on NVIDIA Ada Lovelace and Hopper architectures (L4, H100 GPUs).