Cloudflare outage caused by bad software deploy (updated)
Jul 2, 2019John Graham-Cumming
Summary
This post details a global outage caused by a misconfigured WAF Managed Rule during a routine deployment. The rule contained a regular expression that caused CPU utilization to spike to 100% across the network, resulting in 502 errors for customers. The incident highlighted insufficient testing processes for WAF rules, leading to a global deployment in simulated mode that failed to catch the issue. The fix involved a 'global termination' of the ruleset, followed by rollback, testing, and re-enabling of the WAF Managed Rulesets. Changes to testing and deployment processes are being implemented to prevent future incidents.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack