Cloudflare incident on October 30, 2023

What this post added

This post details a critical incident where a misconfiguration in a new deployment tool for Workers KV caused a 37-minute outage. The incident highlighted the dependency of many Cloudflare services on Workers KV and the challenges of performing rollbacks when critical internal tooling is also affected. The root cause was a bug in the deployment tool that incorrectly returned staging build GUIDs for production, leading to traffic being routed to an unauthorized version and resulting in HTTP 401 errors. The resolution involved a manual 'break glass' rollback to a known good build. Key follow-up actions include onboarding KV deployments to standardized Workers deployment models with automated impact detection and recovery, ensuring rollback processes work even when Cloudflare Access is down, and adding pre-checks to deployments.