Mix, Don’t Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

This post systematically compares hyperparameter tuning (specifically high weight decay) against data mixing (using English as an auxiliary language for Arabic) across four model scales (150M to 1.43B parameters) over ~1000 pre-training runs. It quantifies the performance boost from mixing, showing it's equivalent to 2-3x target data for validation loss and 2-13x for downstream task accuracy, with gains scaling with model size. It also highlights that target-language validation loss underestimates mixing's value, as it only captures regularization effects and not knowledge diversification. Practical recommendations include prioritizing mixing ratio over hyperparameter tuning and transferring hyperparameters from a small proxy model via μP.