Partial Cloudflare outage on October 25, 2022
Oct 26, 2022John Graham-Cumming

Summary
Introduced a bug in the Tiered Cache system by adding distributed tracing logic that cleared control headers, leading to 530 DNS errors for requests missing hostname information. This caused approximately 5% of requests to fail at peak. The incident was resolved by rolling back the faulty component. Remediation steps include earlier integration of larger data centers in release plans, expanded acceptance test coverage for Tiered Cache topologies, more aggressive alerting for requests lacking full context, and ensuring systems fail fast during development.