Systemd Network Configuration and Outage Root Cause Analysis

2 posts · May 2023 – Jun 2023

What this trail covers

Datadog experienced a major platform-level incident on March 8, 2023, affecting all services across multiple regions. The incident was triggered by a systemd security patch that restarted systemd-networkd. This restart caused systemd-networkd to flush IP rules it did not manage, leading to network connectivity loss for nodes running Kubernetes control planes and application workloads. The recovery process involved rebooting parent control plane nodes, then child control plane pods, and finally a coordinated reboot of all affected nodes.

All Datadog trails

Jun 20232023-03-08 incident: A deep dive into the platform-level recovery | DatadogThis post details the recovery process following a platform-level incident on March 8, 2023. It explains the hierarchical Kubernetes cluster architecture (parent/child clusters) and the impact of a systemd-networkd restart on network connectivity. The recovery steps involved rebooting parent control plane nodes, then child control plane pods, and finally application nodes. It also details the challenges encountered during scaling, specifically hitting the maximum number of VM instances in a peering group and IP address exhaustion in subnets for log and trace processing clusters. The post provides graphs illustrating the recovery timeline and IP space utilization.
Jun 20232023-03-08 incident: A deep dive into our incident response | DatadogThis post provides a detailed account of the March 8, 2023, global outage, focusing on the incident response process, its successes and failures during the event, and lessons learned. It outlines Datadog's incident response framework, including monitoring strategies (internal and external), the role of the incident commander, communications lead, and customer liaison, and the importance of regular training and a blameless culture. The post then dives into the specific timeline of the March 8 incident, from the systemd upgrade trigger to full restoration, highlighting the initial diagnosis, the identification of Kubernetes failure as the root cause, and the mitigation and recovery steps. It also discusses the challenges faced, such as intermittent failures of the incident management tool and the need for flexibility and creativity in responding to unprecedented system failures.
May 2023Not just another network latency issue: How we unraveled a series of hidden bottlenecks | DatadogThis post details the investigation and resolution of a series of hidden network bottlenecks within Datadog's usage estimation service. The investigation uncovered four distinct issues: a misconfiguration in the network proxy (Envoy) leading to CPU throttling, a Linux kernel bug affecting the Elastic Network Adapter (ENA) that mapped traffic to a single transmit queue, and a subsequent issue related to network proxy misconfiguration during high traffic periods. The team utilized system-level metrics and network path inspection to diagnose and resolve these problems, ultimately improving latency and reducing alert fatigue.
May 20232023-03-08 incident: A deep dive into the platform-level impact | DatadogThis post details the root cause of the March 8, 2023, platform-level incident, which was a systemd security patch that restarted systemd-networkd. This restart caused systemd-networkd to flush IP rules it did not manage, leading to network connectivity loss for nodes running Kubernetes control planes and application workloads. The investigation traced the issue back to changes in systemd-networkd behavior introduced in v248 and v249, which were present in Ubuntu 22.04 but not in the older Ubuntu 20.04. The incident was exacerbated by Datadog's progressive rollout strategy and the specific configuration of unattended upgrades, which meant not all nodes were impacted simultaneously. The post also explains how Kubernetes networking plugins manipulate host routes and rules, and how this interaction with the systemd-networkd behavior led to the outage.
Advertisement