Internationalization and Language Modeling
Integrating autoconversion: Facebook’s path from Zawgyi to Unicode

Integrating autoconversion: Facebook’s path from Zawgyi to Unicode

9/27/2019 · Nick LaGrow, Miri Pruzan

What this post added

This post details the technical implementation of font converters in Facebook and Messenger to support Myanmar's transition from Zawgyi to Unicode. It explains the challenges of content encoding detection (differentiating Zawgyi from Unicode using ML models trained on Facebook content), device encoding detection (using string width on-device), and the conversion process itself. It also highlights the use of Google's myanmar-tools library and the need to train models on Facebook-specific content due to its informal nature.

Read the original post ↗