Making Facebook self-healing: Automating proactive rack maintenance

This post introduces Aggregate Maintenance Handlers, an enhancement to the FBAR service, to automate rack-level and multi-rack maintenance. It details the decision-making process based on maintenance impact (scope, type, time, duration) and outlines a four-step handler disable/enable process (preflight check, pre-disable, host-level disable, post-disable). It also introduces Dapper for coordinating automated and manual maintenance, and discusses lessons learned from serial and parallel use of disable logic, highlighting the need for smarter, aggregated approaches to avoid capacity issues and race conditions.