Cross-lingual pretraining sets new state of the art for natural language understanding

Introduced a new cross-lingual pretraining approach using Translation Language Modeling (TLM) and Masked Language Modeling (MLM) objectives. This method concatenates parallel sentences and trains models to infer masked words by attending to text in either or both languages. Achieved state-of-the-art results in unsupervised and supervised machine translation, and cross-lingual text classification, especially for low-resource languages. Released code and pretrained models to the community.