Control Plane Resilience

3 posts · Apr 2020 – May 2026

What this trail covers

Cloudflare's control plane and analytics services experienced an outage due to a catastrophic failure at a third-party data center provider. While many core services remained online due to a robust high availability cluster design, critical systems like Kafka and ClickHouse, powering analytics, had tight dependencies on services exclusively running in the affected facility. This incident highlighted the need to formally require all new products and their associated databases to integrate with the global network and ensure resilience against single points of failure.

All Cloudflare trails

May 2026Code Orange: Fail Small is complete. The result is a stronger Cloudflare networkThis post details the 'Code Orange' initiative, a comprehensive effort to enhance Cloudflare's infrastructure resilience following major outages. Key technical contributions include: 1. Snapstone: A new internal component for health-mediated deployment of configuration changes, enabling gradual rollouts, real-time health monitoring, and automated rollbacks. 2. Failure mode review: Product teams have reviewed and implemented 'fail stale' or 'fail open' strategies for critical services, removing non-essential runtime dependencies. 3. System segmentation: Independent service copies for different customer cohorts (e.g., free customers first for Workers runtime) to limit blast radius. 4. Revised 'break glass' procedures: Development of backup authorization pathways for 18 key services with emergency scripts and proxies, and engineering-wide drills. 5. Internal Codex: A mandatory system with AI-powered agents for enforcing engineering standards and preventing regressions, derived from RFCs and incident learnings.
Dec 2025Code Orange: Fail Small — our resilience plan following recent incidentsThis post details Cloudflare's "Code Orange: Fail Small" resilience plan following two recent network incidents. The plan focuses on three key areas: 1) Implementing controlled rollouts for all configuration changes propagated to the network, mirroring existing practices for software binary releases. This involves treating configuration updates with the same tested caution as code changes, using a Health Mediated Deployment (HMD) system. 2) Reviewing and improving failure modes of all systems handling network traffic to ensure well-defined behavior under unexpected error states, including assuming failure between service interfaces and implementing sane defaults. 3) Changing internal "break glass" procedures and removing circular dependencies to enable faster access to systems during incidents while maintaining security. The post highlights that configuration changes, powered by Quicksilver, propagated instantly and caused the outages, necessitating the introduction of controlled deployments within Quicksilver. It also discusses how assuming failure at service interfaces, such as between Bot Management and core network software, could have prevented widespread impact.
Dec 2025Code Orange: Fail Small — our resilience plan following recent incidentsThis post details Cloudflare's "Code Orange: Fail Small" initiative, a comprehensive plan to enhance network resilience following two recent major incidents. The initiative introduces controlled rollouts for configuration changes, mirroring the process for software releases, to prevent rapid propagation of errors. It also emphasizes reviewing and improving inter-service interface contracts to ensure graceful failure handling with sane defaults, and overhauling "break glass" procedures and removing circular dependencies to expedite emergency response. The plan aims to implement these changes iteratively, with key goals for configuration management, failure mode adherence, and emergency access to be met by the end of Q1.
Nov 2025Cloudflare outage on November 18, 2025This post details a specific outage caused by a database permission change leading to an oversized feature file for Bot Management, which exceeded processing limits in the core proxy. It explains the failure mode: a doubled feature file size due to duplicate rows from a ClickHouse query, exceeding a fixed-size limit in the Bot Management software. The post also describes the recovery process: stopping the bad file propagation, replacing it with a good version, and restarting core proxies. It notes differential impact on FL and FL2 proxy engines and the coincidence of the status page outage. The root cause is traced to a change in ClickHouse query behavior affecting the generation of the feature file.
Sep 2025A deep dive into Cloudflare’s September 12, 2025 dashboard and API outageIntroduced a bug in the dashboard's React useEffect hook dependency array that caused repeated, unnecessary calls to the Tenant Service API. This, combined with a service update to the Tenant Service API, overwhelmed the service and led to an outage. The incident also highlighted the lack of Argo Rollouts for the Tenant Service and insufficient capacity allocation for the service. Improvements are planned to add retry information to API calls, spread out retries with random delays, and increase resource allocation for the Tenant Service.
Advertisement
Aug 2025Cloudflare incident on August 21, 2025This post details a network congestion incident caused by a single customer's traffic surge overwhelming links between Cloudflare and AWS us-east-1. It explains how AWS's BGP prefix withdrawals exacerbated the issue, leading to packet loss and high latency. The incident was resolved through manual traffic engineering by both Cloudflare and AWS. Remediations include developing traffic deprioritization mechanisms, expediting DCI upgrades, improving BGP traffic engineering coordination with AWS, and building a long-term per-customer network resource management system.
Jun 2025Cloudflare service outage June 12, 2025This post details a significant service outage caused by a third-party cloud provider failure impacting Cloudflare's Workers KV service. It outlines the cascading failures across numerous Cloudflare products (Workers KV, Access, Gateway, WARP, Dashboard, Challenges, Turnstile, Browser Isolation, Images, Stream, Realtime, Workers AI, Pages & Workers Assets, AutoRAG, Durable Objects, D1, Queues & Event Notifications, AI Gateway, CDN, Workers / Workers for Platforms, Workers Builds, Browser Rendering, Zaraz) due to their reliance on Workers KV. The post highlights the failure mode of Workers KV (cold reads and writes failing due to a central data store outage) and the impact on specific services, including error rates and user-facing disruptions. It also mentions the mitigation strategy of activating kill switches for Challenges and Turnstile, and the ongoing transition of Workers KV to more resilient infrastructure. The incident emphasizes the critical need for robust dependency management and architectural resilience, especially when relying on third-party infrastructure.
Mar 2025Cloudflare incident on March 21, 2025This post details a specific incident where a human error during a credential rotation process for the R2 Gateway service led to a temporary authentication failure with the storage backend. The error involved inadvertently deploying new credentials to a development instance instead of production due to omitting the `--env` parameter in Wrangler commands. This resulted in 100% write failures and ~35% read failures for R2, impacting various dependent services. The incident lasted over an hour because of a lack of visibility into which credentials were being used by the Gateway Worker. The post outlines the incident timeline, the impact on different services, and the root cause. It also implicitly highlights the need for better automation and visibility in credential management and deployment processes to prevent such human errors.
Feb 2025Cloudflare incident on February 6, 2025This post details a 59-minute outage on February 6, 2025, impacting R2 object storage and dependent services due to human error during an abuse remediation action. The R2 Gateway service was inadvertently disabled, leading to a 100% failure rate for R2 operations and cascading failures in services like Stream, Images, Cache Reserve, Vectorize, and Log Delivery. The incident was caused by insufficient validation safeguards in admin tooling, which allowed a product disablement action to impact a production service. The root cause was identified by reviewing service deployment history and configuration. Remediation involved escalating to an operations team with lower-level system access to re-enable the R2 Gateway service and triggering a redeployment. The incident also caused a temporary increase in error rates for Durable Objects due to client reconnections. The post emphasizes the importance of system-level controls to reduce the blast radius of any action that could disable production services and highlights that no data was lost or corrupted.
Sep 2024Removing uncertainty through "what-if" capacity planningThis post details the development and implementation of 'Scenario Planner,' a system designed for "what-if" capacity planning in Cloudflare's global network. It models demand and supply changes across data centers, converting various units (e.g., server count, customer growth multipliers) into a common unit of 'CPU Time' (milliseconds per second per request). The system outputs heatmaps and 'Expected Failovers' views to visualize capacity projections and inform infrastructure decisions, aiming to proactively address potential performance impacts and ensure availability.
Jun 2024Cloudflare incident on June 20, 2024This post details an incident where a new DDoS mitigation rule exposed a latent bug in the rate limiting system, causing processes to enter an infinite loop due to a combination of improper tail calls in Lua and a broken cookie validation function. This led to a significant increase in latency and error rates. The incident response involved identifying the root cause, testing service restarts, and ultimately disabling the new DDoS rule. Changes were made to prevent recurrence, including improving the rate limiting system's key generation and handling of multiple cookies.
Nov 2023Post mortem on the Cloudflare Control Plane and Analytics OutageThis post details a significant outage impacting Cloudflare's control plane and analytics services caused by a catastrophic power failure at a third-party data center (PDX-04). It outlines the intended design of a three-data center active-active high availability cluster for the control plane, the failure of this design due to unexpected dependencies of critical analytics systems (Kafka, ClickHouse) on services exclusively running in the affected facility, and the lack of testing for full data center outages. The post also describes the sequence of events leading to the outage, including the data center provider's power issues, UPS battery failure, and communication breakdowns. Key learnings include the need for stricter integration requirements for new products into the HA cluster and comprehensive testing of full data center failures.
Jan 2023Cloudflare incident on January 24, 2023This post details a specific incident where a code release error in service token management led to a 121-minute outage affecting various Cloudflare services, including Workers, Zero Trust, and CDN control plane functions. The root cause was identified as an incorrect update to service token metadata, specifically overwriting the client secret with an empty string due to a bug in the 'last seen at' field implementation. The incident timeline, the technical explanation of service tokens, the impact on specific products like WARP and Zero Trust, and the remediation steps (manual restoration and database backup restore) are documented. The post also highlights the interdependencies within Cloudflare's infrastructure ('Cloudflare runs on Cloudflare') as a key factor in the widespread impact.
Nov 2020A Byzantine failure in the real worldThis post details a real-world incident where a Byzantine fault in a network switch led to cascading failures in Cloudflare's control plane. It describes the timeline of events, including etcd errors due to conflicting information from a partially failed switch, automatic promotion of a primary database replica which overloaded the primary due to a lack of available replicas, and the impact on the dashboard's user session management. The post highlights the challenges of modeling and anticipating failures in systems with redundancy where components enter a degraded state. It also discusses the engineering tradeoffs between simpler protocols like RAFT and Byzantine Fault Tolerance (BFT) for cluster management systems, and the need for BFT solutions suitable for general-purpose services like etcd.
Apr 2020Cloudflare Dashboard and API Outage on April 15, 2020This post details an outage caused by the disconnection of multiple fibre connections from a core data center during planned maintenance. The incident resulted in the unavailability of the Cloudflare Dashboard and API, impacting configuration changes, cache purging, and other management functions. The response involved failing over to a disaster recovery data center and restoring connectivity. Moving forward, Cloudflare plans to improve design by spreading external connectivity across multiple parts of the facility, enhance documentation for quick identification of critical cables, and update processes to clearly call out cabling that should not be touched during hardware retirement.