TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Model Quantization for Inference Optimization

9 posts · May 2026 – Sep 2026

What this trail covers

This feature thread tracks the evolution of model quantization techniques to optimize AI model inference performance. Initial efforts focused on the general concepts and benefits of quantization, followed by post-training quantization (PTQ) methods using tools like NVIDIA Model Optimizer. This post details the process of converting FP8-quantized checkpoints into high-performance TensorRT inference engines, including ONNX export, TensorRT engine building, and performance profiling, demonstrating significant speedups. Subsequent developments introduce NVFP4, a 4-bit mixed-precision training recipe for JAX and MaxText on NVIDIA Blackwell and Rubin platforms, achieving up to 1.73x speedup over FP8 baselines with negligible accuracy loss by leveraging subbyte precision and native hardware support.

All NVIDIA trails
  1. High-Throughput Structure Prediction with BioNeMo Inference Runtime | NVIDIA Technical BlogThis post introduces the NVIDIA BioNeMo Inference Runtime (BioIR) for accelerating biomolecular structure-prediction models on NVIDIA GPUs, preserving PyTorch workflows. It details BioIR's end-to-end processor and direct PyTorch module integration, highlighting optimizations at the kernel, CUDA Graph, and Ray pipeline scaling levels. Benchmarks demonstrate significant throughput and energy efficiency gains for structure prediction tasks.
  2. How to Size GPUs for AI Inference and TCO Without Overspending | NVIDIA Technical BlogThis post details practical strategies for sizing GPU infrastructure for AI inference workloads and optimizing Total Cost of Ownership (TCO). It introduces a framework for mapping inference workloads to specific use-case categories (AI Chatbots/Copilots, AI Agents, Content Generation, Translation Apps) and outlines key sizing inputs such as model selection, DAUs, concurrency, input/output string lengths, cache hit rate, and latency targets. It also presents a core-and-flex capacity model and emphasizes the importance of right-sizing GPUs to workload memory footprints and latency targets. Furthermore, it highlights model optimization techniques like FP8 post-training quantization (demonstrated with NVIDIA Model Optimizer), pruning, and knowledge distillation as methods to shrink memory footprint and lower TCO. Sample scenarios for financial services and life sciences are provided to illustrate these concepts.
  3. Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect | NVIDIA Technical BlogThis post introduces NVIDIA TensorRT Model Connect, a new reference implementation project that bridges open models from Hugging Face checkpoints to native C++ inference using TensorRT. It details a two-phase deployment workflow: building a bundle from a model ID via a Python CLI, and then loading and running that bundle in a C++ application. It highlights two API levels (semantic and module-level) for flexibility and the integration of custom GPU kernels via TVM FFI. The project emphasizes AI-native development with coding agents and nightly releases to accelerate support for the rapidly evolving open model ecosystem, aiming to deliver faster inference than torch.compile and a consistent deployment path for supported open models.
  4. Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer | NVIDIA Technical BlogThis post details the application of Quantization-Aware Distillation (QAD) using NVIDIA Model Optimizer to develop the Nemotron 3.5 Lightning NVFP4 checkpoint. It describes the two-stage QAD process: first, post-training quantization (PTQ) to W4A16 for the student model, targeting a 95-99% median accuracy recovery to leave room for QAD. Second, quantization-aware distillation where the student is trained against a frozen BF16 teacher using KL divergence loss and simulated quantization. The post highlights that QAD consistently outperforms PTQ in accuracy recovery and agentic/coding benchmarks, enabling more aggressive quantization (e.g., NVFP4) that would otherwise cause unacceptable degradation. It also provides specific PTQ recipes and discusses the end-to-end workflow reproduction using NVIDIA Model Optimizer and Megatron-Bridge.
  5. Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer | NVIDIA Technical BlogThis post details the process of creating the NVIDIA Nemotron 3 Ultra NVFP4 checkpoint using NVIDIA Model Optimizer. It explains the challenges of FP4 quantization, including the limited representable values and the impact of outliers. The post contrasts 'max scaling' (absmax) and 'mean squared error (MSE) scaling' for determining quantization scales, highlighting their respective limitations. It introduces 'four-over-six scaling' as a method to address the gap between representable FP4 values (4 and 6) by analyzing error distribution and optimizing for the specific quantization grid. The post also discusses mixed precision quantization, where different layers are quantized to different formats (NVFP4, FP8, BF16, FP16) based on sensitivity and accuracy impact, and how the checkpoint adapts to different hardware (Hopper vs. Blackwell).
  7. Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding | NVIDIA Technical BlogIntroduces DFlash, an open-source block diffusion model for speculative decoding that accelerates LLM inference by drafting entire token blocks in parallel. Demonstrates up to 15x throughput improvement on NVIDIA Blackwell GPUs for gpt-oss-120b and nearly doubles interactivity for Llama 3.1 8B compared to EAGLE-3. Highlights integration with SGLang, vLLM, and TensorRT-LLM, enabling adoption without code refactoring.
  8. How to Optimize Transformer-Based Models for Low-Precision Training | NVIDIA Technical BlogThis post details how to optimize transformer-based models for low-precision training by moving from high-level model settings to concrete GEMM workloads. It introduces a microbenchmarking tool that converts transformer configurations and batch sizes into specific GEMM shapes, enabling empirical benchmarking across BF16, MXFP8, and NVFP4 precisions. The post emphasizes profiling Fprop, Dgrad, and Wgrad separately to account for aspect ratio effects and kernel dispatch differences. It contrasts autocast mode (which includes quantization cost) with pre-quantizing (which isolates raw GEMM kernel performance) to understand quantization overhead. Empirical results on ESM2-15B demonstrate that while large GEMMs benefit significantly from NVFP4 over MXFP8, smaller GEMMs benefit less, and theoretical hardware speedups are reduced in practice by quantization, block scaling, and stochastic rounding overheads.
  9. Model Quantization: Turn FP8 Checkpoints into High-Performance Inference Engines with NVIDIA TensorRT | NVIDIA Technical BlogThis post details the practical steps for converting FP8-quantized CLIP checkpoints into high-performance TensorRT inference engines. It covers exporting the ModelOpt checkpoint to ONNX format, highlighting the resulting ~34-50% reduction in ONNX file size due to folding Q-DQ nodes. It also describes profiling the FP8 TensorRT engine using `trtexec` and Nsight Deep Learning Designer, demonstrating a 1.39x-1.45x speedup in inference latency for CLIP encoders compared to FP16, attributed to optimized FP8 matmul and attention kernel execution.
  10. Train Models Faster with JAX and MaxText Using NVFP4 on NVIDIA Blackwell | NVIDIA Technical BlogThis post introduces the NVFP4 training recipe for JAX (implemented in MaxText) on NVIDIA Blackwell and Rubin platforms. It details five core techniques (micro block scaling, E4M3 block scale factors, selective Random Hadamard Transform, 2D FP8 scaling, and stochastic rounding) that preserve convergence in large-scale LLM training while achieving up to 1.73x speedup over FP8 baselines. The NVFP4 format leverages subbyte precision and native hardware support on the GB300 Grace Blackwell Ultra Superchip for high-throughput GEMM operations in transformer MLP layers, while attention blocks remain at higher precision to avoid softmax quantization noise amplification. The post provides an example script for enabling NVFP4 in MaxText and presents performance results for Llama 3 8B and Llama 3.1 405B models.
  11. NVIDIA Blackwell Sets STAC-AI Record for LLM Inference in Finance | NVIDIA Technical BlogThis post details the STAC-AI LANG6 benchmark results for LLM inference in finance, specifically highlighting the performance of NVIDIA Blackwell GPUs (HGX B200 and RTX PRO 6000) compared to Hopper (GH200). It showcases the use of NVFP4 quantization for Blackwell and FP8 for Hopper, both optimized with TensorRT LLM. The results demonstrate significant throughput and latency improvements, with HGX B200 achieving up to 2.8x performance uplift over GH200 in batch mode. It also touches upon interactive mode metrics like reaction time and words per second, and the importance of chat template application during inference.