Making Facebook self-healing: Automating proactive rack maintenance

What this post added

This post introduces Aggregate Maintenance Handlers, an enhancement to the Facebook Auto Remediation (FBAR) service, designed to safely automate maintenance on multiple servers at once at the rack or multi-rack level. It details the framework for disabling and enabling servers in bulk, considering scope, maintenance type, start time, and duration to make impact-based decisions. The post also describes the four-step disable/enable process (preflight check, pre-disable, host-level disable, post-disable) and introduces Dapper as a tool to coordinate automated and manual maintenance processes, especially when automation fails or is not possible. Lessons learned from serial and parallel use of disable logic are also discussed, highlighting the importance of robust automation for large-scale infrastructure maintenance.