Data Platform Explained Part II

This post details the evolution of Spotify's Data Platform, focusing on Data Collection and Data Management & Processing. For Data Collection, it highlights the scale of over 1 trillion events per day and the ongoing evolution of its event delivery architecture, referencing previous posts on its inception and migration. It also mentions the automated deployment of event-specific components (PubSub queues, anonymization pipelines, streaming jobs) using K8 operators based on event schema definitions. For Data Management & Processing, it discusses the management of over 38,000 scheduled pipelines, execution on BigQuery, Flink, or Dataflow, and the common use of Scio. It also details the management of data endpoints with retention policies, access controls, lineage tracking, and quality checks, deployed via custom K8 operators. Monitoring and Backstage integration are also highlighted. The post also emphasizes building a culture around the data platform through documentation, onboarding, automated migrations, and community support.