Maximize AI Factory Energy Efficiency Through Full-Stack Inference and Training Optimizations | NVIDIA Technical Blog

This post details how NVIDIA maximizes AI factory energy efficiency through full-stack inference and training optimizations. It highlights system co-design with power, cooling, and infrastructure, and collaboration with ecosystem partners. Key technical contributions include the use of the NVIDIA GB200 NVL72 rack-scale system with direct-to-chip liquid cooling and in-rack power smoothing, NVIDIA DSX for dynamic power allocation and real-time telemetry, and the adoption of narrow precision formats like NVFP4 for improved throughput and energy efficiency. For LLM training, it discusses energy-aware techniques such as coordinated GPU speed tuning to minimize idle time and reduce energy consumption without increasing training time, and fine-grained profiling of kernel and phase-level energy usage in collaboration with the ML.ENERGY Initiative and Megatron-LM.