TechPrufer
Get started
Spotify
Spotify logoSpotify · Feature trail

Perimeter Reliability and Incident Response

12 posts · May 2013 – Apr 2025

What this trail covers

Spotify's perimeter reliability has been enhanced through improved incident response, bug fixes in Envoy filters and Kubernetes memory limits, better configuration rollout strategies, and improved monitoring capabilities. This aims to prevent future outages caused by similar cascading failures. This post details an incident caused by a DNS resolver failure triggered by maintenance on GitHub Enterprise, highlighting the need for improved resiliency against DNS outages and the planned decommission. This post details an outage caused by a bug in a custom Envoy filter triggered by a change in filter order, leading to Envoy crashes and a memory limit misconfiguration with Kubernetes. Mitigation involved increasing perimeter capacity. Future improvements include fixing the bug, correcting the configuration mismatch, improving rollout strategies, and enhancing monitoring.

All Spotify trails
  1. Incident Report: Spotify Outage on April 16, 2025This post details an outage caused by a bug in a custom Envoy filter triggered by a change in filter order, leading to Envoy crashes and a memory limit misconfiguration with Kubernetes. Mitigation involved increasing perimeter capacity. Future improvements include fixing the bug, correcting the configuration mismatch, improving rollout strategies, and enhancing monitoring.
  2. Incident Report: Spotify Outage on January 14, 2023This post details a specific incident on January 14, 2023, where a DNS resolver failure, triggered by scheduled maintenance of GitHub Enterprise, led to a Spotify outage. The incident was caused by a new failure mode in the DNS component that applied invalid configurations when GHE was brought back online. The post outlines the timeline of the incident, the root cause, and the mitigation steps. It also details future plans to fix the bug, improve resiliency against DNS outages, and decommission the problematic DNS component.
  3. Failing Forward — How We Grow from IncidentsThis post analyzes incidents affecting Spotify for Artists in 2021, quantifying their impact on productivity and preventability. It highlights that most incidents have a moderate to high productivity impact, with 23% affecting more than one responder for a significant portion of a day. The analysis categorizes incidents by preventability, with most scoring 3 or higher, indicating localized failures that could be prevented or mitigated. A key finding is the significant reduction in Time to Recover (TTR) for incidents where synthetic tests were in place, showing a 10x faster recovery time. The post also discusses the challenges in accurately recording incident timelines and the need for better systems and incentives to capture this data, referencing DORA metrics.
  4. Incident Report: Spotify Outage on March 8, 2022This post details a global outage on March 8, 2022, caused by an outage in Google Cloud Traffic Director and a bug in a gRPC client library. The incident led to users being unable to log back into Spotify apps. The immediate mitigation involved reverting affected systems to use DNS-based service discovery. Short-term actions include working with Google Cloud to understand the root cause and adding more monitoring and alerting for service discovery issues. The long-term investment is in further resiliency through monitoring, automatic error detection, and self-recovery.
  5. Four Lessons We Learned from Creating Spotify’s Desktop AppThis post details the historical development of Spotify's desktop app, focusing on the re-architecture of its peer-to-peer (P2P) networking. Key technical decisions included moving away from browser-based limitations to a standalone 'fat client' and building custom protocols to hybridize client-server and P2P technology. This was driven by the need for improved speed and user experience compared to music piracy, and required a full-stack infrastructure build from the ground up. The post also touches on the 'magic trick' of instant playback as a differentiator.
  7. Techbytes: What The Industry Misses About Incidents and What You Can DoThis post details an incident caused by a DNS resolver failure triggered by maintenance on GitHub Enterprise, highlighting the need for improved resiliency against DNS outages and the planned decommission of legacy systems. It emphasizes understanding incidents through Resilience Engineering techniques, moving beyond traditional DevOps approaches to a broader and deeper understanding of online software worlds.
  8. Automated Incident Response Infrastructure in GCPThis post details the deployment of Google's GRR (GRR Rapid Response) framework for automated incident response and live forensics within Spotify's GCP infrastructure. It describes the architecture of GRR (Frontend, Worker, Admin UI) and the use of Terraform to deploy a scalable, secure instance. Key technical contributions include the use of Container-Optimized OS, GCP Load Balancers, Identity-Aware Proxy for authentication, and Google SQL for communication. The post highlights the ability to remotely pull process memory, perform signature searches with Yara, and integrate artifacts into the intrusion detection pipeline, significantly improving visibility and response capabilities for security incidents.
  9. Smoother Streaming with BBRThis post details an A/B experiment with BBR, a new TCP congestion control algorithm, which resulted in increased download bandwidth and decreased stutter for Spotify users, particularly in regions with higher packet loss. The experiment involved enabling BBR on a subset of audio servers and measuring playback latency, stutter, and bandwidth. Results showed a 6-10% decrease in stutter and a 5-15% increase in bandwidth, with more significant improvements in Asia-Pacific and Latin America. A specific incident with an upstream provider highlighted BBR's resilience to network congestion.
  10. Spotify’s Love/Hate Relationship with DNSThis post details Spotify's internal DNS infrastructure, including their use of BIND for authoritative nameservers and Unbound for resolvers and on-host caching. It describes their automated DNS record generation and deployment process, which evolved from manual edits to a cron-based system using Python scripts and Git. The post also introduces Nameless, an internal service discovery system for SRV and A records, and outlines their monitoring strategies for DNS infrastructure, including metrics from Unbound.
  11. ELS: a latency-based load balancer, part 2This post, part 2 of the ELS load balancer series, details the metrics used for load balancing (success latency, success rate, outstanding requests, failure latency, parallelism) and how they are measured using forward exponential decay. It explains the calculation of expected latency from the user's perspective, considering retries and client round-trip time, and how this expected latency is converted into a weight for probabilistic load distribution. It also introduces a tweak to the formula inspired by C3 for Cassandra to improve tail latency by penalizing machines with too many outstanding requests.
  12. Designing the Spotify perimeterThis post details the initial design and implementation of Spotify's network perimeter, focusing on web load balancers and proxy systems. It describes the transition from a chaotic, unmanaged network to a structured perimeter with self-service capabilities for developers. Key technical details include the use of dyn.com for DNS failover, nginx for web load balancing (with HAProxy for active health checks), and Puppet for configuration management. It also touches on multi-tenancy for different traffic types (websites, APIs, payments) and the security implications of exposing services.
  13. Incident Management at SpotifyThis post details a major incident caused by a legacy desktop client bug that caused excessive retries to the Popcount service, leading to an outage. The incident was exacerbated by a new feature dependency and excessive logging. Remediations included fixing the client's retry behavior, implementing fast-fail logic in Popcount, improving logging strategies by separating critical messages and introducing rate limiting, and implementing static caching for dependent services.