Boosting MoE Training Throughput with Advanced Fusion Kernels | NVIDIA Technical Blog

Introduces custom fused MLP kernels for MoE models, built with the CuTe DSL, to address activation, CPU boundedness, and quantization costs. These kernels achieve 1.3x-2x kernel-level speedup by enabling sync-free MoE execution with full-iteration CUDA graphs. They support GLU activation functions (SwiGLU, GeGLU, sReLU) with fused epilogues, handle quantization (MXFP8, NVFP4) and other operations like clamping and bias addition within the fused kernel, and eliminate CPU synchronization overhead by tracking tokens per group within GPU memory. Integration with cuDNN Frontend, Transformer Engine, and Megatron Core leads to up to 8% end-to-end throughput improvement in DeepSeek-V3 and 93% in GPT-OSS pre-training.