Multilingual Knowledge Transfer via Lexical Intervention

This post introduces LINK, a data-level intervention method to improve cross-lingual knowledge transfer during model pretraining. By performing lexical substitutions in the high-resource (English) training corpus using bilingual vocabularies, LINK enhances knowledge acquisition for downstream tasks in low-resource languages without requiring additional model training or complex translation systems. This method has shown significant improvements on downstream tasks across eight languages and five model sizes, achieving up to a 2x speedup in training to reach equivalent performance.