Boosting MoE Training Throughput with Advanced Fusion Kernels | NVIDIA Technical Blog

What this post added

Introduced advanced fused MLP kernels for MoE models, custom-built with the NVIDIA CuTe DSL, to address activation bottlenecks, CPU boundedness, and quantization costs. These kernels achieve 1.3x-2x kernel-level speedup over unfused paths by eliminating memory and synchronization overhead, enabling sync-free MoE execution with full-iteration CUDA Graphs. They support GLU activation functions (SwiGLU, GeGLU, sReLU) by repacking weights for in-epilogue computation and natively handle quantization (MXFP8, NVFP4), feature scaling, tensor clamping, and bias addition. Integration with cuDNN Frontend, Transformer Engine, and Megatron Core leads to up to 8% end-to-end throughput improvement in DeepSeek-V3 and 93% in GPT-OSS pre-training.