2023-03-08 incident: A deep dive into the platform-level impact | Datadog

This post details the root cause of the March 8, 2023, platform-level incident, which was a systemd security patch that restarted systemd-networkd. This restart caused systemd-networkd to flush IP rules it did not manage, leading to network connectivity loss for nodes running Kubernetes control planes and application workloads. The investigation traced the issue back to changes in systemd-networkd behavior introduced in v248 and v249, which were present in Ubuntu 22.04 but not in the older Ubuntu 20.04. The incident was exacerbated by Datadog's progressive rollout strategy and the specific configuration of unattended upgrades, which meant not all nodes were impacted simultaneously. The post also explains how Kubernetes networking plugins manipulate host routes and rules, and how this interaction with the systemd-networkd behavior led to the outage.