That time we unplugged a data center to test our disaster readiness

This post details the culmination of over a year's work by the Disaster Readiness (DR) team to physically unplug the San Jose data center and test disaster preparedness. It highlights the architectural evolution from a centralized San Jose data center to a multi-homed, active-active Magic Pocket for block storage, and the challenges in achieving an active-active metadata stack due to MySQL replication lag, read-committed isolation, and complex data ownership in Edgestore. The post also describes the creation of the dedicated DR team and the development of modular, safe failover tooling, inspired by Facebook's Maelstrom paper, to address issues identified during a previous outage.