Foundation Model Training Scalability

5 posts · May 2026 – Jun 2026

What this trail covers

Pinterest has achieved near-linear training scalability for its foundation models by addressing communication bottlenecks in distributed training. Initial efforts focused on enabling AWS EFA for OS-bypass networking. Subsequent optimizations included Quantized Communications (QComms) to reduce payload size, Balanced Sharding to ensure even workload distribution, Bandwidth-Aware Embedding Optimization by reshaping embedding dimensions, and a refined 2D Parallel topology (All-to-All optimized) to improve communication efficiency. This has led to significant engagement gains across Pinterest's recommendation surfaces.

All Pinterest trails

Jun 2026Achieving Near-Linear Training Scalability for Pinterest’s Foundation ModelsThis post details the journey to achieve near-linear training scalability for Pinterest's foundation models. It quantifies the initial poor scaling (0.2x at 2 nodes without EFA, 1.13x at 2 nodes and 1.21x at 4 nodes with EFA) and attributes the bottleneck to distributed embedding collective communication. The key optimizations introduced are: 1. Quantized Communications (QComms) to compress FP32 to FP8 for reduced payload size. 2. Balanced Sharding to match hash partitions to GPUs for even workload distribution. 3. Bandwidth-Aware Embedding Optimization by halving embedding dimension and doubling row count to reduce data movement. 4. A refined 2D Parallel topology (All-to-All optimized) that keeps expensive all-to-all operations local to nodes and uses all-reduce for inter-node synchronization. These optimizations led to 2.0x scaling at 2 nodes and 3.9x at 4 nodes, and 7.5x at 8 nodes.
Jun 2026Achieving Near-Linear Training Scalability for Pinterest’s Foundation ModelsThis post details the optimization journey to achieve near-linear training scalability for Pinterest's foundation models. It describes five key optimizations: 1. Quantized Communications (QComms) to reduce embedding tensor payload size by compressing to FP8. 2. Balanced Sharding to match hash partitions to GPUs for even workload distribution. 3. Bandwidth-Aware Embedding Optimization by reshaping embedding dimensions to halve data shipped per All-to-All operation. 4. 2D Parallel (AllReduce Optimized) topology to keep expensive embedding communication local to each node. 5. 2D Parallel (All-to-All Optimized) topology, flipping the previous approach to keep All-to-All entirely within the node and sync replicas via AllReduce across nodes. These optimizations collectively improved 2-node scaling from 1.13x to 2.0x and 4-node scaling from 1.21x to 3.9x (97.5% of ideal), extending to 8 nodes with a 7.5x scaling factor.
May 2026Making User-Sequence Data More Cost-Efficient, Faster, and Easier to UseThis post details the redesign of Pinterest's user-sequence platform to improve cost-efficiency, speed, and ease of use. Key contributions include: 1. Establishing a 'one definition, many runtimes' principle for signal and event type definitions, ensuring consistency across real-time indexing, batch indexing/backfill, and online serving. 2. Implementing configuration-as-code for sequences and enrichments using Python with a defined schema, validated and compiled into a portable JSON format. 3. Introducing a shared execution engine for filtering, enrichment, and transformation, powering both streaming and batch pipelines. 4. Developing a real-time indexer for low-latency updates and a batch indexer/backfill pipeline for historical data. 5. Utilizing columnar, time-partitioned storage for efficient data access. 6. Providing an online serving API for requesting user sequences. The focus is on providing a stable contract for downstream consumers, improving cost-efficiency, and accelerating the onboarding of new event types and enrichments.
May 2026Enhancing Ad Relevance: Integrating Real-Time Context into Sequential Recommender ModelsIntroduced a Contextual Sequential Two Tower Model to integrate real-time context into ad relevance. This involved adding a context layer to the query tower, using synthetic augmented data for training by injecting pseudo-context from positive labels, and implementing a hybrid inference flow where the Transformer encoder is inferred offline and the context layer/MLP head are computed online. This resulted in a 3x-10x increase in Recall@K, a ~275-300% increase in median candidate relevance on Related Pins, and a ~0.7% lift in ROAS.
May 2026Optimizing ML Workload Network Efficiency (Part I): Feature TrimmerThis post details the implementation of the "Feature Trimmer" system to address network bandwidth bottlenecks in Pinterest's online ML serving root-leaf architecture. It introduces the concept of using model signatures as the source of truth for feature requirements, synchronizing these signatures between model deployment and root configurations to send only necessary features. This significantly reduces root-leaf network traffic, enabling infrastructure cost savings.
Advertisement