TechPrufer
Sign inGet started

Blogs›Datadog›Systemd Network Configuration and Outage Root Cause Analysis

Systemd Network Configuration and Outage Root Cause Analysis

Systemd Network Configuration and Outage Root Cause Analysis

2
posts
2023

This post details the root cause analysis of a major platform-level incident on March 8, 2023, which affected all Datadog services across multiple regions. The incident was triggered by a systemd security patch that restarted systemd-networkd. This restart, on Ubuntu 22.04 hosts (which use a newer systemd version), caused systemd-networkd to flush IP rules it did not manage. This behavior, combined with Datadog's Kubernetes networking configuration (where pods receive IPs from the underlying network), led to a global outage. The post also discusses Datadog's incident response process, highlighting its strengths and weaknesses during this event, and emphasizes the importance of a blameless culture and continuous improvement.

2023

2023-03-08 incident: A deep dive into our incident response | Datadog

6/1/2023

This post provides a deep dive into the root cause analysis of the March 8, 2023 global outage, identifying a systemd security patch restarting systemd-networkd as the trigger. It details how this restart flushed IP rules on Ubuntu 22.04 hosts, interacting with Datadog's Kubernetes networking configuration to cause the outage. The post also elaborates on Datadog's incident response process, including monitoring, high-severity incident handling, practice, and blameless culture, and provides a timeline of the incident response.

2023-03-08 incident: A deep dive into the platform-level impact | Datadog

5/24/2023

This post provides a deep dive into the platform-level impact of the March 8, 2023 incident. It traces the root cause to a systemd-networkd behavior change introduced in systemd v248 and v249, which flushes unknown IP rules upon startup. This behavior was triggered by a systemd security patch applied to Ubuntu 22.04 hosts. The post details how this change interacted with Datadog's Kubernetes networking setup, which uses Cilium and assigns pod IPs from the underlying network, leading to the loss of host routes and rules essential for pod connectivity. It also explains Datadog's progressive rollout strategy for OS updates and how it influenced the scope of the outage.