Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer | NVIDIA Technical Blog

This post details the process of creating the NVIDIA Nemotron 3 Ultra NVFP4 checkpoint using NVIDIA Model Optimizer. It explains the challenges of FP4 quantization, including the limited representable values and the impact of outliers. The post contrasts 'max scaling' (absmax) and 'mean squared error (MSE) scaling' for determining quantization scales, highlighting their respective limitations. It introduces 'four-over-six scaling' as a method to address the gap between representable FP4 values (4 and 6) by analyzing error distribution and optimizing for the specific quantization grid. The post also discusses mixed precision quantization, where different layers are quantized to different formats (NVFP4, FP8, BF16, FP16) based on sensitivity and accuracy impact, and how the checkpoint adapts to different hardware (Hopper vs. Blackwell).