1.5x faster MoE training with custom MXFP8 kernels

This post details the implementation of custom MXFP8 kernels for Mixture-of-Experts (MoE) training on Blackwell GPUs. It addresses challenges related to tensor memory (TMEM) latency and the dominance of quantization overhead. The authors rewrote the MoE layer from scratch using pure CUDA and PTX, achieving a 3.5x improvement in MoE layer performance and a 1.5x end-to-end training speedup on Blackwell GPUs. Key contributions include a detailed explanation of MXFP8 quantization, analysis of dequantization bottlenecks on Blackwell, and a recipe for efficient FP8 quantization kernels.