TechPrufer
Get started
Meta
Meta logoMeta · Feature trail

Internationalization and Language Modeling

9 posts · Sep 2009 – May 2022

What this trail covers

Meta's advancements in internationalization and language processing have evolved from developing a community-driven translation application to enabling non-English characters in fan page URLs, which positively impacted search rankings. This also includes leveraging user contributions for site translations, exploring efficient real-time communication protocols like MQTT for applications like Facebook Messenger, developing a multilingual composer to allow users to write posts in multiple languages, and creating parallel corpora for NLP research. The current post details the use of a community-driven translation app to manage translations for dynamic content with named entities, employing glossaries, dynamic explosion based on token value features, and linguistic rules for inflection. It also discusses the creation of over 250 million individual language models capturing user dialects for applications like spam detection and speech recognition.

All Meta trails
  1. Language packs: Meta’s mobile localization solutionThis post details the development of a custom mobile localization infrastructure at Meta, featuring downloadable language packs and the FBT (Facebook Translation) framework. It addresses challenges with native Android/iOS localization by supporting gendered text and reducing app size. The post explains the end-to-end flow, construction of language packs, FBT string creation with extraction and structure, and the client-side adoption of language packs, including improvements for poor network conditions like prefetching and fallback mechanisms.
  2. Integrating autoconversion: Facebook’s path from Zawgyi to UnicodeThis post details the technical implementation of font converters for Facebook and Messenger to handle the transition from Zawgyi to Unicode encoding in Myanmar. It explains the challenges of content encoding detection (using an ML model trained on Facebook content), device encoding detection (by measuring string width on-device), and the conversion process itself. The post highlights the integration of Google's open-source myanmar-tools library and the challenges of applying this at Facebook's scale across various content types and delivery mechanisms.
  3. Expanding automatic machine translation to more languagesThis post details the expansion of Facebook's automatic machine translation (NMT) services to include 24 new languages. It describes the technical challenges of low-resource language translation, including lack of training data and the need for fast training. The post outlines three key strategies employed: 1) Increasing labeled in-domain data by manually labeling millions of words in 25 languages, showing an average BLEU score increase of +7.2. 2) Exploring semisupervised NMT techniques like back-translation (average +2.5 BLEU), copy-target (average +2.7 BLEU), and mining translations from multilingual webpages (average +0.4 BLEU), along with in-domain data up-sampling for improved fluency and relevance. 3) Utilizing multilingual modeling by combining related language directions, which improved BLEU scores by an average of +4.6. The post also mentions the use of PyTorch Translate, ONNX, and Caffe2 environments for training and production.
  4. Scaling neural machine translation to bigger data sets with faster training and inferenceThis post details significant advancements in scaling neural machine translation (NMT) by drastically reducing training and inference times. Key contributions include achieving a 45x speedup in training through optimizations like 16-bit precision, cumulative updates, and overlapping communication with backward passes. It also highlights the ability to train more accurate models using larger datasets, including monolingual data via back-translation, and improvements in translation speed. The post emphasizes the open-sourcing of the fairseq library to enable faster NMT training for the wider research community.
  5. Under the hood: Multilingual embeddingsIntroduced multilingual embeddings for NLP tasks, enabling text classification across languages without explicit translation. This involves training language-specific embeddings using fastText, projecting them into a common space using an orthogonal matrix derived from parallel data, and integrating them into the DeepText framework. This approach significantly improves cross-lingual accuracy and reduces latency compared to translation-based methods.
  7. Transitioning entirely to neural machine translationThis post details the significant transition from phrase-based statistical machine translation to neural machine translation (NMT) for all of Facebook's backend translation systems. It highlights the use of sequence-to-sequence LSTMs with attention for improved context handling and accuracy, techniques for managing unknown words and vocabulary reduction, and the scaling of these models using Caffe2. The post also introduces the application of convolutional neural networks (CNNs) for translation and the ongoing exploration of multilingual models, marking a major leap in translation quality and efficiency.
  8. Building a better way to write posts in multiple languagesIntroduced a multilingual composer for Pages and individuals, allowing a single post to be written in multiple languages. Implemented machine translation for pre-filling content in additional languages. Redesigned storage to use 'author translations' (separate TAO objects for each language) instead of concatenation. Developed common interfaces for validation logic and caching of multilingual post metadata. Implemented server-side language selection for viewers based on preferences, locale, and posting language, defaulting to the first message with a "See Translation" option.
  9. Meet a Facebook Engineer: Ben GertzfieldThis post details the work on enabling non-English characters in fan page URLs, which previously resulted in garbage URLs for non-English names. It also discusses the success of crowd-sourced translation driven by user engagement and the application of MQTT for real-time messaging in Facebook Messenger.
  10. Names in translation and social language modelingThis post details the technical implementation of a community-driven translation application for handling dynamic content with named entities. It introduces three key systems: Glossary for consistent vocabulary, Dynamic Explosion to split strings based on token value variations (e.g., gender, number), and Linguistic Rules to automatically adjust spelling based on orthographic or phonological rules. It also mentions the creation of over 250 million individual language models for user dialects and their applications in spam detection and speech recognition.