GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

This post details Meta's co-design efforts to double the end-to-end (E2E) training efficiency of its LLM-scale ads foundation model, GEM, achieving 20-25% Model FLOPs Utilization (MFU) and scaling training FLOPs 4x in 12 months. Key innovations include compute efficiency through a customized recommendation kernel library (Jagged Flash Attention - JFA, Generalized Dot-Product Attention - GDPA, BlockAttention) and mixed ultra-low precision training (MXFP8 attention and MLP), and scaling efficiency via topology-aware 5D parallelism with SM-free collectives (2D FSDP + Expert Parallelism for dense parameters, Fully Sharded 2D Model Parallelism for sparse parameters) co-designed with Meta's multi-tiered network hierarchy. JFA specifically addresses jagged inputs by eliminating padding waste and optimizing backward passes, achieving 40-140% TFLOPS improvement over previous versions. GDPA unifies and accelerates diverse attention modules, addressing performance gaps in real-world workloads compared to synthetic benchmarks.