We reverse-engineered Flash Attention 4

This post details the reverse-engineered architecture of Flash Attention 4 (FA4), focusing on its asynchronous pipeline of operations and optimizations for Nvidia's Blackwell architecture. It explains the 'life of a tile' through the GPU memory hierarchy, including the roles of shared memory, Tensor Memory, and the use of CUDA Cores for faster approximate exponentials and more efficient online softmax. The post highlights the dual nature of its asynchronous execution model compared to CPU-based async programming and the use of warp specialization for concurrency within program instances.