TechPrufer
Get started
Datadog
Datadog logoDatadog · Feature trail

Graceful Degradation and Reliability Engineering

4 posts · May 2023 – Oct 2025

What this trail covers

Datadog is evolving its approach to reliability by prioritizing graceful degradation over preventing all failures. This involves implementing persistent intake storage to prevent data loss, making live data available faster by skipping backlogs and prioritizing critical telemetry, and updating retry logic to avoid overwhelming downstream systems. The company is also addressing architectural bottlenecks and technical debt, and introducing prioritization at the infrastructure and compute level to ensure linear degradation of throughput with available compute capacity, preventing complete service failure during resource constraints. This is exemplified by the design of Courier, a new message queuing service, which incorporates multi-tenancy, at-least-once delivery, graceful degradation, and horizontal scalability, utilizing formal modeling, lightweight simulations, and chaos testing to validate its design.

All Datadog trails
  1. Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog | DatadogThis post details Datadog's shift in reliability strategy from 'never-fail' architectures to 'failing better' by prioritizing graceful degradation. Key technical changes include implementing disk-based persistent intake storage to prevent data loss, prioritizing real-time data processing over backlogs, updating retry logic to use different backends and backoff mechanisms, and introducing prioritization at the infrastructure and compute level (e.g., Kubernetes PriorityClass, faster autoscaler) to ensure critical services remain available during outages.
  2. How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems | DatadogThis post details the application of formal modeling (TLA+) and lightweight simulations to the design of Courier, a new multi-tenant message queuing service. It highlights how these techniques were used to validate design decisions related to multi-tenancy (sharding tenants across FoundationDB clusters), at-least-once delivery, graceful degradation (linear throughput reduction with compute capacity), and horizontal scalability. The design incorporates lessons learned from a past outage, aiming to prevent complete service failure during resource constraints. The post also mentions the use of chaos testing as a complementary validation method.
  3. 2023-03-08 incident: A deep dive into the platform-level recovery | DatadogThis post details the platform-level recovery process following a major incident, focusing on the EU1 region. It describes the phased restoration of Kubernetes control planes (parent and child clusters) and application nodes, highlighting the importance of rebooting nodes to regain network connectivity. It also details the challenges encountered when scaling clusters to process backlogged data, specifically hitting limits on the maximum number of VM instances in a peering group and the maximum number of IPs available in subnets for high-traffic clusters. The post explains how these limits delayed full recovery and the steps taken to address them, including engaging with the cloud provider.
  4. Not just another network latency issue: How we unraveled a series of hidden bottlenecks | DatadogThis post details the investigation and resolution of a series of network latency issues impacting the usage estimation service. The investigation uncovered bottlenecks related to Envoy proxy CPU utilization during deployments, leading to TCP retransmits. It also identified a Linux kernel bug affecting the Elastic Network Adapter (ENA) on AWS, which caused traffic to be mapped to a single transmit queue instead of distributing it across multiple queues. The solutions involved allocating more CPU to Envoy and applying a hotfix for the Linux kernel bug, ultimately improving latency during both normal operations and deployments.