Post mortem on the Cloudflare Control Plane and Analytics Outage

What this post added

This post details a major outage of Cloudflare's control plane and analytics services caused by a power failure at a third-party data center. It highlights the failure of a critical data center provider, the unexpected behavior of backup power systems, and the lack of communication from the provider. Crucially, it reveals that while the core network remained operational, critical control plane and analytics systems (Kafka, ClickHouse) were not fully integrated into the high availability cluster and had non-obvious dependencies on the failed data center, leading to extended downtime. The post outlines intended design for data center failure, what failed (dependencies on PDX-04, incomplete HA cluster migration, lax GA requirements for new products), what worked (core network, disaster recovery facility for some services), and proposed changes to ensure all critical systems are in the HA cluster and dependencies are better managed.