Spotify

Incident Report: Spotify Outage on April 16, 2025

May 1, 2025Spotify Engineering
Incident Report: Spotify Outage on April 16, 2025
SummaryAI, from the source post

This post details an outage caused by a bug in a custom Envoy filter triggered by a change in filter order, leading to Envoy crashes and a memory limit misconfiguration with Kubernetes. Mitigation involved increasing perimeter capacity. Future improvements include fixing the bug, correcting the configuration mismatch, improving rollout strategies, and enhancing monitoring.

More from the Perimeter Reliability and Incident Response trail