How to Optimize Transformer-Based Models for Low-Precision Training | NVIDIA Technical Blog

This post details how to optimize transformer-based models for low-precision training by moving from high-level model settings to concrete GEMM workloads. It introduces a microbenchmarking tool that converts transformer configurations and batch sizes into specific GEMM shapes, enabling empirical benchmarking across BF16, MXFP8, and NVFP4 precisions. The post emphasizes profiling Fprop, Dgrad, and Wgrad separately to account for aspect ratio effects and kernel dispatch differences. It contrasts autocast mode (which includes quantization cost) with pre-quantizing (which isolates raw GEMM kernel performance) to understand quantization overhead. Empirical results on ESM2-15B demonstrate that while large GEMMs benefit significantly from NVFP4 over MXFP8, smaller GEMMs benefit less, and theoretical hardware speedups are reduced in practice by quantization, block scaling, and stochastic rounding overheads.