Site Reliability and Outage Management
Power Loss Siren: Making Meta resilient to power loss events

Power Loss Siren: Making Meta resilient to power loss events

12/16/2021 · Raghunathan Modoor Jagannathan, Sulav Malla, Parimala Kondety

What this post added

Introduced the Power Loss Siren (PLS), a rack-level, low-latency distributed power loss detection and alert system. PLS leverages existing in-rack batteries to notify services about impending power loss at least 45 seconds in advance, allowing for proactive mitigation handlers. The architecture includes PLS Relay on rack switches for detection and PLS Handler on servers for mitigation. This system improves reliability for single-primary databases by enabling proactive promotion of remote secondaries and for web servers by stopping new requests and initiating graceful shutdowns, reducing peak error rates by 100x. PLS also simplifies physical infrastructure management by reducing reliance on dual-powered rows and buffer server capacity.

Read the original post ↗