Mixture Pretraining Scaling Laws

This post introduces a repetition-aware mixture scaling law for language model pretraining under data constraints. It accounts for the decreasing value of repeated target tokens and the regularizing role of generic data, providing principled recommendations for effective mixture configurations. The work studies the trade-off between target data repetition and generic data regularization across various model and dataset sizes, finding that repetition is a central driver of target-domain performance and that mixture training tolerates higher repetition than single-source training.