Using Apache Spark for large-scale language model training

What this post added

This post details the migration of a large-scale, complex (100+ stage) N-gram language model training pipeline from HQL over Hive to Apache Spark. It highlights the use of Spark's imperative DSL and RDD interface to gain finer control over data distribution, avoid data skew, and implement application-specific optimizations. Key challenges addressed include managing large '0-shards' for smoothing, potential job instability due to data skew, and executor/shuffle service OOM errors. The solution involved a progressive sharding approach, iteratively refining data distribution based on N-gram history length (1-word, 2-word, etc.) and dynamically adjusting shard sizes to ensure balanced workloads. The use of Spark's `pipe()` operator facilitated the integration of existing C++ binaries, enabling a smooth transition without re-implementing core logic.