Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions

Introduces LINK, a data-level intervention method for multilingual knowledge transfer during pretraining. LINK involves lexical substitutions in the high-resource (English) corpus using bilingual vocabularies. This approach requires no additional model training and only a bilingual vocabulary. Evaluation on eight languages and five model sizes demonstrates improvements in downstream tasks and up to a 2x training speedup to reach equivalent performance.