Reproducing OLMo 3 7B Pre-training in MaxText: case study of large scale training on TPUs- Google Developers Blog

This post details the reproduction of OLMo 3 7B pre-training using MaxText on Google Cloud TPUs. Key contributions include: PyTorch to JAX model conversion with logit-parity checks, verification of a data-loader bug using held-out evaluations, demonstration of checkpoint-and-resume reliability over multi-week runs and after host failures, successful mid-flight resizing of training jobs, seamless switching of TPU generations (Ironwood to v5p), performance optimizations achieving 44.5% MFU on Ironwood, and co-design of attention mechanisms for improved TPU utilization and speed. The post also outlines the necessary MaxText components built for this reproduction, including the model architecture, optimizer, data pipeline, and checkpoint conversion tools.