Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models

This post details the journey to achieve near-linear training scalability for Pinterest's foundation models. It quantifies the initial poor scaling (0.2x at 2 nodes without EFA, 1.13x at 2 nodes and 1.21x at 4 nodes with EFA) and attributes the bottleneck to distributed embedding collective communication. The key optimizations introduced are: 1. Quantized Communications (QComms) to compress FP32 to FP8 for reduced payload size. 2. Balanced Sharding to match hash partitions to GPUs for even workload distribution. 3. Bandwidth-Aware Embedding Optimization by halving embedding dimension and doubling row count to reduce data movement. 4. A refined 2D Parallel topology (All-to-All optimized) that keeps expensive all-to-all operations local to nodes and uses all-reduce for inter-node synchronization. These optimizations led to 2.0x scaling at 2 nodes and 3.9x at 4 nodes, and 7.5x at 8 nodes.