Lights Out, Systems On: Validating Instant Power Loss Readiness

This post introduces 'Instantaneous PowerLoss Storm,' a new testing paradigm for validating zero-notice power loss readiness in Meta's data center regions. It details the defense-in-depth strategies implemented across the DC stack, including battery backup, Power Loss Siren (PLS), and unavailability events (UEs) for the Twine orchestrator. Key challenges addressed include bootstrapping an entire region, specifically circular dependencies among control plane services (solved with Belljar tests and a Twine recovery kit) and the 'boomerang' effect where UEs impacted orchestrators (mitigated by allowing control plane services to ignore power-related UEs). Tradeoffs were defined, distinguishing between critical impacts (data loss, facility damage) and tolerable risks (transient errors, bounded staleness). Validation is incremental, progressing from pre-production to production regions, with the goal of handling region loss as seamlessly as sub-regional fault domains.