Making Facebook Self-Healing

Introduced FBAR (Facebook Auto-Remediation), a system that automates the remediation of server and software outages. FBAR consists of an Alert Fetch Loop to query monitoring systems, a Job Queue, and a Job Engine that executes remediation plugins. These plugins interact with infrastructure APIs to perform actions like removing servers from production, flagging hardware for replacement, and re-enabling repaired machines. FBAR currently manages over 50% of Facebook's infrastructure and has significantly increased service reliability.