TechPrufer
Get started
Spotify
Spotify logoSpotify · Feature trail

Spotify Data Platform

23 posts · Apr 2013 – Apr 2024

What this trail covers

Spotify's Data Platform, a large-scale GCP-based system handling over a trillion events daily, continues to evolve. This post details the adoption of Apache Crunch as a Java-based data processing framework on top of Hadoop, replacing the less performant Python-based Hadoop Streaming API. Crunch offers type-safety, higher performance, and higher-level abstractions for data pipelines. The platform also introduced `crunch-lib` for common operations and a unified execution layer with an Inversion of Control. The Data Collection platform handles over 1 trillion events per day, with its event delivery architecture evolving through numerous iterations, including a move to cloud-managed services like Pub/Sub and Dataflow. Data Management and Processing focus on empowering Spotify to utilize data effectively, ensuring data integrity, traceability, searchability, and accessibility. Over 38,000 actively scheduled pipelines handle hourly and daily tasks, executing on BigQuery, Flink, or Dataflow clusters, often utilizing Scio. Data pipelines generate data endpoints with retention policies, access controls, lineage tracking, and quality checks, managed via custom K8 operators. Monitoring includes alerts for data lateness, failing workflows, and endpoints, with Backstage integration for resource management, cost analysis, and quality assurance. The platform emphasizes building a strong community through documentation, onboarding, automated migrations, and a dedicated support channel.

All Spotify trails
  1. Data Platform Explained Part IIThis post details the evolution of Spotify's Data Platform, focusing on Data Collection and Data Management & Processing. For Data Collection, it highlights the scale of over 1 trillion events per day and the ongoing evolution of its event delivery architecture, referencing previous posts on its inception and migration. It also mentions the automated deployment of event-specific components (PubSub queues, anonymization pipelines, streaming jobs) using K8 operators based on event schema definitions. For Data Management & Processing, it discusses the management of over 38,000 scheduled pipelines, execution on BigQuery, Flink, or Dataflow, and the common use of Scio. It also details the management of data endpoints with retention policies, access controls, lineage tracking, and quality checks, deployed via custom K8 operators. Monitoring and Backstage integration are also highlighted. The post also emphasizes building a culture around the data platform through documentation, onboarding, automated migrations, and community support.
  2. Data Platform Explained Part IThis post explains the need for and evolution of Spotify's data platform, highlighting its role in processing 1.4 trillion data points daily. It details triggers for building a data platform, such as democratizing data, satisfying financial reporting, ensuring data quality, facilitating experimentation, and enabling ML. The evolution is described through its core components: data collection, data processing, and data management. It also references the event delivery infrastructure and the Confidence experimentation platform.
  3. Large-Scale Generation of ML Podcast Previews at Spotify with Google DataflowThis post details the technical evolution of Spotify's data processing capabilities by integrating the Podz ML pipeline for generating podcast previews. It describes the migration from a microservice architecture to a scalable, managed pipeline system using Google Dataflow and Apache Beam. Key technical contributions include: the use of custom containers for managing complex ML framework dependencies, optimizing for latency and cost by transitioning from batch to streaming processing with autoscaling, and leveraging the Klio library to achieve a 30x reduction in preview generation latency (from 111.7 minutes to 3.7 minutes). The post also covers the implementation of monitoring and observability for the streaming pipeline.
  4. How We Built Infrastructure to Run User Forecasts at SpotifyThis post details the development of a new infrastructure for running user forecasts at Spotify. The system enables on-demand and automated weekly forecasts with advanced hyperparameter tuning, significantly reducing model training times. It incorporates distinct workstreams for mature markets (time series models), new markets (proxy-based models), and custom/specialty markets requiring human input. The infrastructure leverages Dataflow for parallel processing and Dockerized Python jobs on Kubernetes. Key components for mature market forecasts include a Parameter-Space Creator, Data Labeler, Hyperparameter Tuning, Model Selector, Model Applier, QC & Manual Intervention tools, and a Publish & Visualization module. Learnings highlight the importance of balancing quality control with fast iteration, smart scaling choices, and system simplicity.
  5. Why We Switched Our Data Orchestration ServiceThis post details the migration of Spotify's data orchestration service from Luigi and Flo to Flyte. The migration was driven by the need to reduce maintenance and support burden, enable platform automation, and gain better insights into workflow execution. Flyte was chosen for its extensibility, multi-language support, scalability, and platform approach. The integration involves Flyte Admin for workflow registration and Styx for execution triggering.
  7. Changing the Wheels on a Moving Bus — Spotify’s Event Delivery MigrationThis post details the migration of Spotify's Event Delivery Infrastructure (EDI) from a legacy, stateful, TSV-based system to a modern, cloud-native, Avro-based system. Key technical contributions include: addressing data loss issues by redesigning event delivery for mobile clients, improving control plane UX by schema-aware tooling and faster schema propagation, and migrating away from stateful services and legacy perimeter logging. A data transformation pipeline was implemented to handle legacy clients and enable a gradual transition to the new data model. The strategy involved finding early adopters, iterating on features, and solidifying interfaces to concurrently migrate internal users while building the new EDI.
  8. Listening Together, Miles ApartThis post details the technical implementation of the 'Listening Together' campaign, which visualizes real-time global song plays. It describes the use of Google Cloud Pub/Sub for event ingestion, Apache Beam with Scio (Spotify's Scala API for Beam) and Google Dataflow for scalable stream processing, and windowing features to group plays within short timeframes. IP addresses are converted to locations using MaxMind databases. A backend service built on Spotify's Apollo framework, autoscaled in Kubernetes, pulls sessions from a new Pub/Sub topic to serve the website. The post highlights challenges in global synchronization and the efficiency of serving pre-calculated data.
  9. My beatThis post details the work of the Babka team in consolidating non-royalty-bearing stream counts for external sharing. Key technical efforts include the creation of a new, dynamic database for Spotify's artist analytics, which will introduce some latency to provide more flexibility in data retrieval and ensure users are shown data relevant to their rights. The team is also working on 'provider APIs' for record labels, managing technical queries and feasibility requests.
  10. Spotify’s Event Delivery – Life in the CloudThis post details the evolution of Spotify's Event Delivery Infrastructure (EDI) to a Cloud Pub/Sub-based system on GCP. Key technical achievements include migrating business-critical data first for rapid stability, achieving over 8M events/sec and 350 TB of data daily, and implementing principles like isolation per event type (each with its own Pub/Sub topic and ETL), prioritizing liveness over lateness, and leveraging managed services. Challenges with autoscaling and deployment tooling are also discussed, highlighting the need for improved iteration cycles.
  11. Techbytes: Data Processing with ScioThis post details Spotify's history of data processing and its transition to Scio, an open-source Scala API for Apache Beam and Google Cloud Dataflow. Scio is used to process petabytes of data, offering a more efficient and scalable approach compared to previous methods.
  12. Techbytes: Handling Big Data at SpotifyThis post details the use of Scio for processing big data at Spotify, highlighting its advantages in simplifying certain data processing jobs and outlining the challenges encountered in handling large datasets. It specifically mentions the use of Scio with Big Data.
  13. Scio 0.7: a deep diveThis post details the release of Scio 0.7.0, introducing two major technical advancements: the ScioIO trait for a more structured and testable approach to I/O operations, and the adoption of compile-time Coder derivation using Magnolia to replace runtime Kryo serialization. The ScioIO trait simplifies I/O logic by abstracting away test behavior and providing typed read/write parameters. The move to compile-time Coder derivation with Magnolia addresses issues with Kryo's performance, memory leaks, and opaque error handling, leading to more efficient and reliable data serialization for Apache Beam jobs. These changes have resulted in observed cost reductions and runtime improvements for production workflows.
  14. Scalable User PrivacyThis post details the development and implementation of Padlock, a key management service designed to enforce Spotify's privacy standard of encrypting all personal user data before persistence. Padlock manages user-specific keychains, using derived keys for services to encrypt/decrypt data. It is built for high scale (1M lookups/sec), low latency (p99 < 15ms), and high availability (99.95% SLO), utilizing Cassandra with Memcached caching and specific optimizations like LOCAL_ONE consistency reads and disabled read repair. The system allows for fine-grained data lifecycle management by categorizing data and assigning separate keys per category.
  15. Autoscaling Pub/Sub ConsumersThis post details the implementation of autoscaling for Pub/Sub consumers within Spotify's Event Delivery system. It describes the architecture using dedicated Pub/Sub topics per event type, consumed by independent ETL processes. The Pub/Sub Consumer microservice, built with Apollo, Docker, and deployed via Helios, is managed by regional instance groups with CPU-based autoscaling. Key challenges addressed include handling unhealthy machines (e.g., Docker zombie states) by implementing a custom HTTP healthchecker that directly destroys unresponsive instances, and fine-tuning CPU usage to prevent autoscaler misbehavior during downstream service issues or event backlogs by employing exponential backoffs. The post also discusses future plans to explore Kubernetes for improved bin packing and resource optimization.
  16. Big Data Processing at Spotify: The Road to Scio (Part 1)This post introduces Scio, a Scala API for Apache Beam and Google Cloud Dataflow, designed for building batch and streaming data pipelines at scale. It details Spotify's historical big data processing tools (Luigi, Scalding, Spark, Storm), the challenges faced, and the migration to Google Cloud. It explains the Apache Beam unified model for batch and streaming processing, highlighting its core primitives (ParDo and GBK) and how it treats batch as a special case of streaming. The post also contrasts Scio's programming model and operational modes (fully managed on Dataflow) with Spark and Scalding, emphasizing Scio's advantages in unified batch/streaming API and reduced operational overhead.
  17. Big Data Processing at Spotify: The Road to Scio (Part 2)This post introduces Scio, a Scala API for Apache Beam and Google Cloud Dataflow, designed to simplify data pipeline development. It details Scio's basic concepts, including its core SCollection abstract data type and its idiomatic Scala approach inspired by Scalding and Spark. Unique features highlighted include type-safe BigQuery integration using Scala macros for compile-time case class generation, and a REPL for interactive data exploration. The post also lists various system integrations (Parquet, Cassandra, Elasticsearch, TensorFlow, etc.) and presents use cases and scaling achievements, noting over 1300 production pipelines built by 200+ internal users, with significant batch and streaming job scales.
  18. Reliable export of Cloud Pub/Sub streams to Cloud StorageThis post details the implementation of a reliable ETL process for exporting Cloud Pub/Sub streams to Cloud Storage. It describes the architecture involving stateless 'Consumer' microservices for reliable Pub/Sub consumption, a 'Completionist' service for tracking intermediate files and determining when hourly buckets are safe to close, and a 'Deduper' Crunch batch job for deduplication. The system utilizes Google Cloud services like Pub/Sub, Cloud Storage, Compute Engine, and Cloud SQL, along with Spotify's internal tools like Apollo, Helios, and Styx. The post also discusses the decision to move away from Dataflow due to operational shortcomings and highlights the importance of robust monitoring and deployment processes for streaming systems.
  19. Spydra – Ephemeral Hadoop Clusters Using Google Compute PlatformThis post introduces Spydra, a new open-sourced project from Spotify designed to automate the lifecycle management of ephemeral Hadoop clusters on Google Compute Platform (GCP) Dataproc. Spydra simplifies job submission, enhances troubleshooting of failed jobs, and supports both ephemeral clusters for single job executions and static, long-living clusters. It is a key component of Spotify's data infrastructure migration to GCP, leveraging experience from managing a 2500-node on-premise Hadoop cluster. Spydra enables dual use of on-premise Hadoop and GCP Dataproc during the migration.
  20. Spotify’s Event Delivery – The Road to the Cloud (Part II)This post details the design of Spotify's new event delivery system, which replaces an older log-file-based system. The new system aims for simpler producer logic and relies on a dedicated Event Delivery Service, a Reliable Persistent Queue, and an ETL job. Initially, Kafka 0.8 was chosen for the queue but proved unstable, leading to a switch to Google Cloud Pub/Sub. The post highlights the benefits of Pub/Sub, including global availability, a simple REST API, and managed operations, while also detailing the rigorous testing performed to ensure it could handle Spotify's production load of 2M events per second.
  21. Spotify’s Event Delivery – The Road to the Cloud (Part III)This post details the implementation of an ETL job using Google Cloud Dataflow to export events from Pub/Sub into hourly buckets stored in Cloud Storage and BigQuery. It explains the use of Dataflow's windowing, triggers, and GroupByKey transform for processing streaming data, handling late arrivals, and managing memory constraints. The post also covers monitoring event timeliness and writing data to persistent storage via the IOChannelFactory interface.
  22. Switching user database on a running systemThis post details the migration of Spotify's user database from PostgreSQL to Cassandra. The migration involved a 'darkloading' strategy where production requests were handled in parallel by both databases to ensure capacity and find bugs before the final switch. Existing accounts were migrated via a job that iterated over users, ensuring data synchronization. The switch itself was a configuration change followed by a synchronized restart of services. Challenges encountered included issues with Cassandra's Lightweight Transactions (LWT) and a bug in the Java driver.
  23. Data Processing with Apache Crunch at SpotifyThis post details the adoption and implementation of Apache Crunch as a Java-based data processing framework at Spotify. It highlights the limitations of the previous Python-based Hadoop Streaming API (performance issues, runtime failures, code duplication) and explains the benefits of Crunch, including type-safety, high performance, higher-level abstractions (filters, joins, aggregations), first-class Avro support, and pluggable execution engines (MapReduce, Spark). The post also introduces `crunch-lib` for common data processing operations and a unified execution layer using an Inversion of Control pattern for better integration with scheduling systems. Example code demonstrates how to build data pipelines with Crunch.
  24. Analytics at SpotifyThis post details the evolution of Spotify's analytics pipeline and infrastructure. It highlights the use of syslog parsing, HDFS for data storage, and the availability of data in the Data Warehouse and Dashboards within 24 hours, with some data accessible instantly via Storm. It introduces the internal tools: Dashboards for custom data screens, Data Warehouse for direct data access and Hive Map/Reduce jobs, and Luigi for complex data operations. The post also describes the development of a transactional email system to measure user engagement impact, which involved logging email send events to syslog for subsequent analysis in HDFS.