NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure | NVIDIA Technical Blog

This post details four case studies of performance degradation in AI training infrastructure, identifying specific configuration gaps at the kernel, hypervisor, BIOS, and NCCL levels. It provides diagnostic patterns using tools like perf, Nsight Systems, and NCCL tests to pinpoint issues such as SMMU overhead in virtualized environments (GB200 NVL72), CPU power management and NUMA misbinding (H100 cluster), insufficient NCCL queue-pair concurrency on high-bandwidth fabrics, and missing runtime topology files within containers. The post outlines the fixes applied, including enabling CMDQV for SMMU, optimizing CPU C-states and NUMA bindings, tuning NCCL settings, and ensuring correct topology propagation, leading to significant improvements in training throughput.