How Facebook keeps its large-scale infrastructure hardware up and running
Dec 9, 2020Fred Lin, Harish Dattatraya Dixit, Sriram Sankar
Summary
This post introduces four methodologies for maintaining hardware availability: 1. Automated hardware remediation using MachineChecker, FBAR, and Cyborg. 2. Minimizing performance impact of error reporting by using a hybrid CMCI/SMI mechanism for memory errors. 3. Leveraging ML to predict and prioritize repairs for undiagnosed/misdiagnosed tickets. 4. Automating fleet-level root cause analysis using Scuba and FP-Growth for log analysis.
More from the Site Reliability and Outage Management trail
- Lights Out, Systems On: Validating Instant Power Loss Readiness
- DrP: Meta’s Root Cause Analysis Platform at Scale
- Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and Optimization
- Diff Risk Score: AI-driven risk-aware software development
- How Meta keeps its AI hardware reliable