Incident Report: Spotify Outage on January 14, 2023

This post details a specific incident on January 14, 2023, where a DNS resolver failure, triggered by scheduled maintenance of GitHub Enterprise, led to a Spotify outage. The incident was caused by a new failure mode in the DNS component that applied invalid configurations when GHE was brought back online. The post outlines the timeline of the incident, the root cause, and the mitigation steps. It also details future plans to fix the bug, improve resiliency against DNS outages, and decommission the problematic DNS component.