2023-03-08 incident: A deep dive into our incident response | Datadog

What this post added

This post provides a deep dive into the root cause analysis of the March 8, 2023 global outage, identifying a systemd security patch restarting systemd-networkd as the trigger. It details how this restart flushed IP rules on Ubuntu 22.04 hosts, interacting with Datadog's Kubernetes networking configuration to cause the outage. The post also elaborates on Datadog's incident response process, including monitoring, high-severity incident handling, practice, and blameless culture, and provides a timeline of the incident response.