Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core | NVIDIA Technical Blog

This post details the implementation and optimization of bitwise determinism in NVIDIA Megatron Core for large-scale LLM pretraining. It introduces a workflow for recording ordered per-rank tensor fingerprints to localize and fix nondeterminism, enabling reliable failure replay, loss-spike debugging, system-change validation, and recovery from interrupted training runs. Kernel-level changes (grouped-GEMM writers, fixed reduction order) and validation procedures (granular tracing, GPU tensor fingerprinting) are described, demonstrating a reduction in the determinism tax to approximately 2% at 2,432 GPUs for Nemotron workloads.