FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

This post introduces FlashAttention-4, an algorithm and kernel co-design specifically for asymmetric hardware scaling on Blackwell GPUs. It addresses bottlenecks in SFU units for softmax exponential (forward pass) and shared-memory traffic (backward pass) by implementing new pipelining strategies. Key innovations include a software emulation of the exponential function using FMA units, conditional online softmax rescaling, storing intermediate results in tensor memory for the backward pass, utilizing Blackwell's 2-CTA MMA mode to reduce shared memory traffic and atomic reductions, and a new tile scheduler to handle load imbalance. The post details hardware features of Blackwell (Tensor Memory, fully asynchronous 5th gen tensor cores, 2-CTA MMA) and provides a "feeds and speeds" analysis highlighting the bottlenecks. It achieves up to 1605 TFLOPs/s (71% utilization) on B200 with BF16, outperforming cuDNN and Triton.