Zero-shot transfer across 93 languages: Open-sourcing enhanced LASER library

This post introduces the enhanced LASER (Language-Agnostic SEntence Representations) toolkit, now supporting over 90 languages and 28 alphabets. It achieves massively multilingual sentence representations by embedding all languages in a single shared space, enabling zero-shot transfer of NLP models. Key technical contributions include a shared encoder (five-layer bidirectional LSTM) and decoder, a joint byte-pair encoding (BPE) vocabulary, and training on 223 million sentences of public parallel data. The toolkit achieves state-of-the-art results on cross-lingual natural language inference (XNLI) and parallel corpus mining (BUCC), processing up to 2,000 sentences per second on GPU.