2023-03-08 incident: A deep dive into the platform-level impact | Datadog

What this post added

This post provides a deep dive into the platform-level impact of the March 8, 2023 incident. It traces the root cause to a systemd-networkd behavior change introduced in systemd v248 and v249, which flushes unknown IP rules upon startup. This behavior was triggered by a systemd security patch applied to Ubuntu 22.04 hosts. The post details how this change interacted with Datadog's Kubernetes networking setup, which uses Cilium and assigns pod IPs from the underlying network, leading to the loss of host routes and rules essential for pod connectivity. It also explains Datadog's progressive rollout strategy for OS updates and how it influenced the scope of the outage.