Spotify Unwrapped: How we brought you a decade of data

This post details the engineering challenges and architectural decisions made for the 2019 Spotify Wrapped campaign, focusing on processing a decade's worth of user listening data for over 248 million monthly active users. The User Wrapped data engineering team processed approximately 5x the amount of data compared to the previous year at a fraction of the cost. Key architectural decisions included using Google Cloud Bigtable for efficient time-series data aggregation and as the final data store to reduce shuffles. The system decoupled data processes into smaller 'data stories' for faster iteration and implemented a Python library for direct data access from Bigtable to ensure data quality. The primary takeaway is the importance of leveraging the right system design and data store to reduce cost and decoupling data processes to improve iteration speed.