Spotify’s Event Delivery – The Road to the Cloud (Part II)

This post details the design of Spotify's new event delivery system, which replaces an older log-file-based system. The new system aims for simpler producer logic and relies on a dedicated Event Delivery Service, a Reliable Persistent Queue, and an ETL job. Initially, Kafka 0.8 was chosen for the queue but proved unstable, leading to a switch to Google Cloud Pub/Sub. The post highlights the benefits of Pub/Sub, including global availability, a simple REST API, and managed operations, while also detailing the rigorous testing performed to ensure it could handle Spotify's production load of 2M events per second.