TechPrufer
Sign inGet started

Blogs›Pinterest›Foundation Model Training Scalability

Foundation Model Training Scalability

Foundation Model Training Scalability

2
posts
2026

Pinterest has achieved near-linear training scalability for its foundation models, enabling significant engagement gains across recommendation surfaces. Initial multi-node training was severely bottlenecked by communication overhead, particularly for distributed embedding lookups. Through a series of optimizations including Quantized Communications (QComms), Balanced Sharding, Bandwidth-Aware Embedding Optimization, and a refined 2D Parallel topology (All-to-All optimized), training scalability was improved. This post details the redesign of the user-sequence platform to make sequence data more cost-efficient, faster, and easier to use by defining a signal or event type once and instantiating it consistently across multiple runtimes (real-time indexing, batch indexing/backfill, and online serving). The new platform uses configuration-as-code for sequences and enrichments, a shared execution engine, real-time and batch indexers, columnar time-partitioned storage, and an online serving API to support demanding production use cases for recommendation systems.

2026

Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models

6/25/2026

This post details the journey to achieve near-linear training scalability for Pinterest's foundation models. It identifies the primary bottleneck as distributed embedding collective communication, exacerbated by large embedding tables. The key contributions include: 1. Quantized Communications (QComms) to reduce payload size by compressing FP32 to FP8. 2. Balanced Sharding to ensure even workload distribution across GPUs. 3. Bandwidth-Aware Embedding Optimization by reshaping embedding tables (halving dimension, doubling rows) to reduce data transfer. 4. A refined 2D Parallel topology, specifically the 'All-to-All Optimized' approach, which localizes expensive all-to-all operations within nodes and uses cheaper all-reduce for inter-node synchronization. These optimizations collectively improved 2-node scaling from 1.13x to 2.0x and 4-node scaling from 1.21x to 3.9x, extending to 8 nodes at 7.5x.

Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

5/21/2026

This post details the redesign of Pinterest's user-sequence platform to improve cost-efficiency, speed, and ease of use. Key contributions include: 1. Establishing a 'one definition, many runtimes' principle for signal and event type definitions, ensuring consistency across real-time indexing, batch indexing/backfill, and online serving. 2. Implementing configuration-as-code for sequences and enrichments using Python with a defined schema, validated and compiled into a portable JSON format. 3. Introducing a shared execution engine for filtering, enrichment, and transformation, powering both streaming and batch pipelines. 4. Developing a real-time indexer for low-latency updates and a batch indexer/backfill pipeline for historical data. 5. Utilizing columnar, time-partitioned storage for efficient data access. 6. Providing an online serving API for requesting user sequences. The focus is on providing a stable contract for downstream consumers, improving cost-efficiency, and accelerating the onboarding of new event types and enrichments.