Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

This post details the redesign of Pinterest's user-sequence platform to improve cost-efficiency, speed, and usability. Key contributions include: 1. **Configuration-as-Code:** Defining sequence and enrichment logic in Python, compiled to JSON, for faster onboarding and improved reviewability. 2. **Shared Execution Engine:** A unified engine for both streaming and batch processing that reuses enrichment logic, minimizing code duplication and drift. 3. **Pluggable Executors:** Modular 'business logic' for specific event types within the shared engine. 4. **Unified Runtimes:** A 'one definition, many runtimes' approach ensuring consistency across training, batch, and serving. 5. **Columnar, Time-Partitioned Storage:** Optimized storage for efficient reads and writes. The platform aims to provide a stable contract for downstream ML consumers.