2023-03-08 incident: A deep dive into our incident response | Datadog

This post provides a detailed account of the March 8, 2023, global outage, focusing on the incident response process, its successes and failures during the event, and lessons learned. It outlines Datadog's incident response framework, including monitoring strategies (internal and external), the role of the incident commander, communications lead, and customer liaison, and the importance of regular training and a blameless culture. The post then dives into the specific timeline of the March 8 incident, from the systemd upgrade trigger to full restoration, highlighting the initial diagnosis, the identification of Kubernetes failure as the root cause, and the mitigation and recovery steps. It also discusses the challenges faced, such as intermittent failures of the incident management tool and the need for flexibility and creativity in responding to unprecedented system failures.