Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog | Datadog

This post details Datadog's shift in reliability strategy from 'never-fail' architectures to 'failing better' by prioritizing graceful degradation. Key technical changes include implementing disk-based persistent intake storage to prevent data loss, prioritizing real-time data processing over backlogs, updating retry logic to use different backends and backoff mechanisms, and introducing prioritization at the infrastructure and compute level (e.g., Kubernetes PriorityClass, faster autoscaler) to ensure critical services remain available during outages.