Incident Report: Spotify Outage on April 16, 2025
May 1, 2025Spotify Engineering

Summary
This post details an outage caused by a bug in a custom Envoy filter triggered by a change in filter order, leading to Envoy crashes and a memory limit misconfiguration with Kubernetes. Mitigation involved increasing perimeter capacity. Future improvements include fixing the bug, correcting the configuration mismatch, improving rollout strategies, and enhancing monitoring.