Graceful Degradation and Reliability Engineering

10/15/2025

This post details Datadog's response to a large outage by shifting focus from preventing all failures to enabling graceful degradation. Key technical contributions include implementing disk-based persistent intake storage to prevent data loss, optimizing systems to skip processing backlogs and prioritize live data, updating retry logic with backoff mechanisms and dead letter queues, and introducing prioritization at the infrastructure and compute level (e.g., PriorityClass in Kubernetes, faster autoscaler, global audit of job priorities). The post also discusses the need to move away from 'never-fail' architectures that prioritize full stop over partial but accurate data, and the importance of serving partial results when full data is unavailable.

11/20/2024

This post details the application of formal modeling (TLA+) and lightweight simulations to the design of Datadog's new message queuing service, Courier. It describes how these techniques were used to verify system-level properties and failure modes, particularly focusing on multi-tenancy, at-least-once delivery, graceful degradation, and horizontal scalability. The design of Courier, which uses multiple FoundationDB clusters for tenant isolation and a broker layer with health checks, was analyzed to ensure it met these requirements, incorporating lessons learned from a previous global outage.

6/16/2023

This post details the platform-level recovery process following a major incident, focusing on the EU1 region. It describes the phased restoration of Kubernetes control planes (parent and child clusters) and application nodes, highlighting the importance of rebooting nodes to regain network connectivity. It also details the challenges encountered when scaling clusters to process backlogged data, specifically hitting limits on the maximum number of VM instances in a peering group and the maximum number of IPs available in subnets for high-traffic clusters. The post explains how these limits delayed full recovery and the steps taken to address them, including engaging with the cloud provider.

5/26/2023

This post details the investigation and resolution of a series of network latency issues impacting the usage estimation service. The investigation uncovered bottlenecks related to Envoy proxy CPU utilization during deployments, leading to TCP retransmits. It also identified a Linux kernel bug affecting the Elastic Network Adapter (ENA) on AWS, which caused traffic to be mapped to a single transmit queue instead of distributing it across multiple queues. The solutions involved allocating more CPU to Envoy and applying a hotfix for the Linux kernel bug, ultimately improving latency during both normal operations and deployments.