TechPrufer
Sign inGet started

Blogs›NVIDIA›Model Quantization for Inference Optimization

Model Quantization for Inference Optimization

Model Quantization for Inference Optimization

5
posts
2026

This feature thread tracks the evolution of model quantization techniques to optimize AI model inference performance. Initial efforts focused on the general concepts and benefits of quantization, followed by post-training quantization (PTQ) methods using tools like NVIDIA Model Optimizer. This post details the process of converting FP8-quantized checkpoints into high-performance TensorRT inference engines, including ONNX export, TensorRT engine building, and performance profiling. Subsequent developments have focused on NVFP4 quantization for Blackwell GPUs, demonstrating significant performance gains on benchmarks like STAC-AI LANG6 for LLM inference in finance, achieving up to 2.8x improvement on HGX B200 compared to GH200.

2026

Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer | NVIDIA Technical Blog

6/26/2026

This post details the process of creating the NVIDIA Nemotron 3 Ultra NVFP4 checkpoint using NVIDIA Model Optimizer. It introduces NVFP4 quantization for NVIDIA Blackwell architecture, achieving higher inference throughput and reduced model size. The post highlights a mixed-precision approach for the Nemotron 3 Ultra checkpoint, using NVFP4, FP8, BF16, and FP16 based on layer sensitivity. It also explains how the checkpoint adapts to different hardware (Hopper and Blackwell) and delves into the challenges and solutions for FP4 quantization, including max scaling, MSE scaling, and a new four-over-six scaling method that optimizes for FP4 representable value gaps to improve accuracy.

Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding | NVIDIA Technical Blog

6/23/2026

Introduces DFlash, an open-source block diffusion model for speculative decoding that accelerates LLM inference by drafting entire token blocks in parallel. Demonstrates up to 15x throughput improvement on NVIDIA Blackwell GPUs for gpt-oss-120b and nearly doubles interactivity for Llama 3.1 8B compared to EAGLE-3. Highlights integration with SGLang, vLLM, and TensorRT-LLM, enabling adoption without code refactoring.

How to Optimize Transformer-Based Models for Low-Precision Training | NVIDIA Technical Blog

6/16/2026

This post details how to optimize transformer-based models for low-precision training using FP8 and NVFP4 on NVIDIA Hopper and Blackwell GPUs. It introduces a microbenchmarking tool that converts model configurations and batch sizes into specific GEMM shapes, enabling empirical benchmarking across different precisions (BF16, MXFP8, NVFP4) and stages (Fprop, Dgrad, Wgrad). The post highlights the importance of analyzing both autocast and pre-quantized results to understand quantization overhead and raw kernel performance, demonstrating that while larger GEMMs benefit significantly, smaller GEMMs may see less improvement due to overheads. It provides a practical example using ESM2-15B to illustrate how to interpret these results for real-world speedup estimation.

Model Quantization: Turn FP8 Checkpoints into High-Performance Inference Engines with NVIDIA TensorRT | NVIDIA Technical Blog

6/9/2026

This post details the practical steps for converting an FP8-quantized CLIP model checkpoint into a TensorRT inference engine. It covers exporting the model to ONNX format with QuantizeLinear/DequantizeLinear (Q/DQ) nodes, explaining how TensorRT fuses these nodes for direct execution on FP8 Tensor Cores. The post also includes benchmarking results using `trtexec` on an NVIDIA RTX 6000 Ada GPU, demonstrating a 1.39x-1.45x speedup in inference latency for FP8 engines compared to FP16, attributed to optimized FP8 matmul and attention kernels.

NVIDIA Blackwell Sets STAC-AI Record for LLM Inference in Finance | NVIDIA Technical Blog

5/27/2026

This post details the STAC-AI LANG6 benchmark results for LLM inference in finance, specifically highlighting the performance of NVIDIA Blackwell GPUs (HGX B200 and RTX PRO 6000) compared to Hopper (GH200). It showcases the use of NVFP4 quantization for Blackwell and FP8 for Hopper, both optimized with TensorRT LLM. The results demonstrate significant throughput and latency improvements, with HGX B200 achieving up to 2.8x performance uplift over GH200 in batch mode. It also touches upon interactive mode metrics like reaction time and words per second, and the importance of chat template application during inference.