Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog | Datadog

What this post added

This post details Datadog's response to a large outage by shifting focus from preventing all failures to enabling graceful degradation. Key technical contributions include implementing disk-based persistent intake storage to prevent data loss, optimizing systems to skip processing backlogs and prioritize live data, updating retry logic with backoff mechanisms and dead letter queues, and introducing prioritization at the infrastructure and compute level (e.g., PriorityClass in Kubernetes, faster autoscaler, global audit of job priorities). The post also discusses the need to move away from 'never-fail' architectures that prioritize full stop over partial but accurate data, and the importance of serving partial results when full data is unavailable.