Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

What this post added

This post details the redesign of Pinterest's user-sequence platform to improve cost-efficiency, speed, and ease of use. Key contributions include: 1. Establishing a 'one definition, many runtimes' principle for signal and event type definitions, ensuring consistency across real-time indexing, batch indexing/backfill, and online serving. 2. Implementing configuration-as-code for sequences and enrichments using Python with a defined schema, validated and compiled into a portable JSON format. 3. Introducing a shared execution engine for filtering, enrichment, and transformation, powering both streaming and batch pipelines. 4. Developing a real-time indexer for low-latency updates and a batch indexer/backfill pipeline for historical data. 5. Utilizing columnar, time-partitioned storage for efficient data access. 6. Providing an online serving API for requesting user sequences. The focus is on providing a stable contract for downstream consumers, improving cost-efficiency, and accelerating the onboarding of new event types and enrichments.