Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

This post introduces Olmo-core 3, a new open-source training infrastructure specifically designed for large Mixture-of-Experts (MoE) models. It details the architectural shift from Fully Sharded Data Parallelism (FSDP) to Distributed Data Parallelism (DDP) for MoE training, which keeps experts resident on GPUs and routes data to them, improving throughput. Key optimizations include row-wise expert parallelism, GPU-resident routing, and grouped GEMM. The post also highlights the benefits of MXFP8 for reducing computation and memory usage, showing a 21% throughput increase and reduced peak memory compared to BF16. Benchmarks demonstrate scaling to trillion-parameter models with high TFLOP/s/GPU. The post also discusses observed failure modes like 'token gerrymandering' and the importance of matching input values for performance comparisons.