Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models

This post details the optimization journey to achieve near-linear training scalability for Pinterest's foundation models. It describes five key optimizations: 1. Quantized Communications (QComms) to reduce embedding tensor payload size by compressing to FP8. 2. Balanced Sharding to match hash partitions to GPUs for even workload distribution. 3. Bandwidth-Aware Embedding Optimization by reshaping embedding dimensions to halve data shipped per All-to-All operation. 4. 2D Parallel (AllReduce Optimized) topology to keep expensive embedding communication local to each node. 5. 2D Parallel (All-to-All Optimized) topology, flipping the previous approach to keep All-to-All entirely within the node and sync replicas via AllReduce across nodes. These optimizations collectively improved 2-node scaling from 1.13x to 2.0x and 4-node scaling from 1.21x to 3.9x (97.5% of ideal), extending to 8 nodes with a 7.5x scaling factor.