Meta

Power Loss Siren: Making Meta resilient to power loss events

Dec 16, 2021Raghunathan Modoor Jagannathan, Sulav Malla, Parimala Kondety
Power Loss Siren: Making Meta resilient to power loss events
SummaryAI, from the source post

Introduced the Power Loss Siren (PLS), a rack-level, low-latency, distributed system for detecting and alerting on impending power loss events. PLS leverages existing in-rack batteries to provide advance notification (45 seconds) to services, enabling proactive mitigation handlers to failover or gracefully shut down before server power is lost. The architecture includes PLS Relay on rack switches for detection and PLS Handler on servers for mitigation. PLS has demonstrated significant improvements in service reliability, reducing peak error rates during power loss events by 100x for web servers and enabling near-invisible failover for single-primary databases. It also simplifies physical infrastructure management by reducing the need for dual-powered rows and associated buffer capacity.

More from the Site Reliability and Outage Management trail