Control Plane Resilience

11/27/2020

This post details a specific incident where a partially failed network switch caused a Byzantine fault in the etcd cluster, leading to cascading failures in database primary promotion and subsequent overload. It highlights the limitations of RAFT in handling Byzantine faults, the impact on database replicas and user sessions, and the manual intervention required to restore service. The post also discusses the ongoing work to address identified limitations in database replica rebuilds, user session flexibility, and auto-remediation configurations, and the ongoing research into Byzantine Fault Tolerance (BFT) for general-purpose services.

4/16/2020

This post details a significant outage of the Cloudflare Dashboard and API caused by the accidental disconnection of multiple fiber connections from a core data center during planned maintenance. The incident highlights a single point of failure in the patch panel connecting external fiber links. The response involved a dual effort to restore connectivity and failover to a disaster recovery data center. Moving forward, Cloudflare plans to address design flaws by distributing connectivity across multiple parts of the facility, improve documentation and labeling of critical cables, and refine operational processes to prevent similar incidents.

8/26/2011

This post details the precautions taken by a data center in Ashburn, VA, to prepare for Hurricane Irene, including generator testing, fuel management, staff arrangements, and physical site security. It also highlights Cloudflare's architectural design for resilience, emphasizing that the service can continue operating even if entire data centers go offline due to its distributed nature and immediate traffic rerouting to other data centers. The 'Always Online™' feature is also mentioned as a safeguard for hosted sites.