How Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020
Feb 1, 2021Neville Li, Claire McGinty, Sahith Nallapareddy, & Joel Östlund
Summary
This post details the design and implementation of Sort Merge Bucket (SMB) joins within Spotify's data pipelines, specifically for optimizing the Wrapped 2020 Dataflow job. It introduces the SortedBucketSink and SortedBucketSource transforms, explains the role of FileOperations and BucketMetadata, and discusses optimizations such as date partitioning, sharding, and parallelism strategies for joining sources with different bucket counts. It also highlights the SortedBucketTransform for efficient read-transform-write operations and improvements to Beam's external sorter. The post details the adoption of SMB by core data producers and its application in Wrapped 2020.
More from the Data Lake Indexing for Point Queries trail
- Indexing the Data Lake for Online Point Queries | Spotify Engineering
- Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering
- It’s All Just Wiggly Air: Building Infrastructure to Support Audio Research
- How We Improved Data Discovery for Data Scientists at Spotify
- Introducing Chartify: Easier chart creation in Python for data scientists