Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine | NVIDIA Technical Blog

This post details the integration of NVIDIA Transformer Engine with JAX to accelerate dropless MoE training. Key contributions include: 1) Achieving a 10.4x throughput improvement for MoE training on NVIDIA GB200 by optimizing DeepSeek-V3 performance from 103 to 1,068 TFLOPS/GPU. 2) Implementing dropless MoE using grouped GEMM kernels that handle variable expert token counts in a single call, preserving model quality without token dropping or padding. 3) Accelerating expert parallelism operations with NCCL EP, which fuses dispatch and combine stages and deduplicates tokens to reduce network traffic. 4) Incorporating additional optimizations like JAX host offloading and XLA multistreaming collectives to reduce memory bottlenecks and overlap communication. 5) Demonstrating sustained 97% scaling efficiency at 1,024 GPUs on NVIDIA GB300 NVL72 hardware for training DeepSeek-V3 671B.