How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song

This talk details the architecture and training of MiniMax M3, focusing on its Sparse Attention (MSA) mechanism and native multimodal capabilities. MSA is presented as a novel sparse attention design that uses an index branch to select relevant blocks and a sparse branch to attend only to those, achieving approximately 9x faster prefill and 15x faster decoding than full attention. The talk explains why existing sparse attention designs like DeepSeek's DSA were not suitable for MiniMax's architecture due to grouped-query attention and GPU memory access patterns. It also covers the benefits of training text and vision modalities from step zero, contrasting it with adding vision later, and discusses the use of 3D attention within the vision transformer for improved visual understanding. The implementation details include block-level retrieval and efficient kernels optimized for GPUs.