Better MoE model inference with warp decode
Apr 6, 2026Less Wright, Federico Cassano & Zhiyuan Zhang

Summary
Introduced 'warp decode', a novel approach to MoE inference kernel design that reorganizes parallelism around individual output values (neurons) rather than experts. This eliminates five bookkeeping stages present in conventional MoE paths, removes intermediate activation gather and per-expert output buffers, and leverages warp independence for improved GPU scheduling and latency hiding. Achieves 1.84x throughput gain and 1.4x closer accuracy to FP32 reference on Blackwell GPUs.