Cross-lingual pretraining sets new state of the art for natural language understanding

What this post added

Introduced a new cross-lingual pretraining approach for NLP tasks, including a translation language modeling (TLM) objective that extends masked language modeling (MLM) by concatenating parallel sentences. This method achieved state-of-the-art results in unsupervised and supervised machine translation, and cross-lingual text classification for low-resource languages. Released code and pretrained models to the community.