Power Loss Siren: Making Meta resilient to power loss events

Introduced the Power Loss Siren (PLS), a rack-level, low-latency, distributed system for detecting and alerting on impending power loss events. PLS leverages existing in-rack batteries to provide advance notification (45 seconds) to services, enabling proactive mitigation handlers to failover or gracefully shut down before server power is lost. The architecture includes PLS Relay on rack switches for detection and PLS Handler on servers for mitigation. PLS has demonstrated significant improvements in service reliability, reducing peak error rates during power loss events by 100x for web servers and enabling near-invisible failover for single-primary databases. It also simplifies physical infrastructure management by reducing the need for dual-powered rows and associated buffer capacity.
More from the Site Reliability and Outage Management trail
- Lights Out, Systems On: Validating Instant Power Loss Readiness
- DrP: Meta’s Root Cause Analysis Platform at Scale
- Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and Optimization
- Diff Risk Score: AI-driven risk-aware software development
- How Meta keeps its AI hardware reliable