TechPrufer
Get started
Spotify
Spotify logoSpotify · Feature trail

Event Delivery System Migration

1 post · Jan 2016

What this trail covers

Spotify is migrating its event delivery system to the cloud, specifically leveraging Google Cloud managed services like Pub/Sub and DataFlow. This post details the architecture and lessons learned from the existing Kafka-based system, highlighting its limitations in terms of latency, data completeness, and flexibility. The new system aims to address these issues by providing more robust and scalable event transportation and processing.

All Spotify trails
  1. Spotify’s Event Delivery – The Road to the Cloud (Part I)This post details the architecture and operational challenges of Spotify's legacy event delivery system, which is based on Kafka 0.7. It highlights issues with data completeness due to the reliance on HDFS as the single point of failure, latency introduced by ETL transformations from text to Avro, and the loss of flexibility in managing different Quality of Service (QoS) levels for event streams. The post also discusses the limitations of the producer-consumer model and the challenges of handling machine failures and missing end-of-file markers. It sets the stage for the upcoming posts that will detail the new cloud-based event delivery system.