Names in translation and social language modeling
Sep 8, 2009डेविड एलिस
Summary
This post details the technical implementation of a community-driven translation application for handling dynamic content with named entities. It introduces three key systems: Glossary for consistent vocabulary, Dynamic Explosion to split strings based on token value variations (e.g., gender, number), and Linguistic Rules to automatically adjust spelling based on orthographic or phonological rules. It also mentions the creation of over 250 million individual language models for user dialects and their applications in spam detection and speech recognition.
More from the Internationalization and Language Modeling trail
- Language packs: Meta’s mobile localization solution
- Integrating autoconversion: Facebook’s path from Zawgyi to Unicode
- Expanding automatic machine translation to more languages
- Scaling neural machine translation to bigger data sets with faster training and inference
- Under the hood: Multilingual embeddings