Cursor

Better MoE model inference with warp decode

Apr 6, 2026Less Wright, Federico Cassano & Zhiyuan Zhang
Better MoE model inference with warp decode
SummaryAI, from the source post

Introduced 'warp decode', a novel approach to MoE inference kernel design that reorganizes parallelism around individual output values (neurons) rather than experts. This eliminates five bookkeeping stages present in conventional MoE paths, removes intermediate activation gather and per-expert output buffers, and leverages warp independence for improved GPU scheduling and latency hiding. Achieves 1.84x throughput gain and 1.4x closer accuracy to FP32 reference on Blackwell GPUs.

More from the MXFP8 Kernel Optimization for MoE Training trail