Foundation Model Training Scalability
Pinterest has achieved near-linear training scalability for its foundation models, enabling significant engagement gains across recommendation surfaces. Initial multi-node training was severely bottlenecked by communication overhead, particularly for distributed embedding lookups. Through a series of optimizations including Quantized Communications (QComms), Balanced Sharding, Bandwidth-Aware Embedding Optimization, and a refined 2D Parallel topology (All-to-All optimized), training scalability was improved. This post details the redesign of the user-sequence platform to make sequence data more cost-efficient, faster, and easier to use by defining a signal or event type once and instantiating it consistently across multiple runtimes (real-time indexing, batch indexing/backfill, and online serving). The new platform uses configuration-as-code for sequences and enrichments, a shared execution engine, real-time and batch indexers, columnar time-partitioned storage, and an online serving API to support demanding production use cases for recommendation systems.
