Partial Cloudflare outage on October 25, 2022

What this post added

This post details an incident where a change to the distributed tracing logic within Cloudflare's Tiered Cache system caused a significant outage. The bug, introduced by wrapping the `go()` function with `trace_fn()`, inadvertently cleared control headers essential for internal DNS lookups, resulting in 530 errors. The incident response involved identifying the root cause, confirming it through a rollback, and executing an accelerated global rollback. Remediation steps include incorporating larger data centers into earlier release stages, expanding acceptance test coverage for Tiered Cache topologies, improving alerting for requests lacking full context, and ensuring systems fail fast in error scenarios.