Scaling Laws for Mixture Pretraining Under Data Constraints

This post introduces a repetition-aware mixture scaling law for language model pretraining under data constraints. It quantifies the optimal number of repetitions for scarce target corpora (15-20x) based on target data size, compute budget, and model scale. The scaling law accounts for the decreasing value of repeated target tokens and the regularizing role of generic data, providing a principled way to compute effective mixture configurations and practical recommendations for pretraining under data constraints.