Spotify’s Event Delivery – The Road to the Cloud (Part I)

This post details the architecture and operational challenges of Spotify's legacy event delivery system, which is based on Kafka 0.7. It highlights issues with data completeness due to the reliance on HDFS as the single point of failure, latency introduced by ETL transformations from text to Avro, and the loss of flexibility in managing different Quality of Service (QoS) levels for event streams. The post also discusses the limitations of the producer-consumer model and the challenges of handling machine failures and missing end-of-file markers. It sets the stage for the upcoming posts that will detail the new cloud-based event delivery system.