BellJar: A new framework for testing system recoverability at scale
May 5, 2022Christopher Bunn, Jie Huang

Summary
Introduced BellJar, a new framework for testing system recoverability at scale. BellJar provides an isolated, vacuum-sealed experimentation environment that simulates total infrastructure outages by default, allowing teams to test recovery strategies and identify dependencies. It uses an additive allowlisting scheme to selectively reintroduce necessary production capabilities, enabling granular testing of system startup procedures and validation criteria. BellJar aims to replace tribal knowledge and documentation with codified, repeatable recovery tests deployable in CI/CD pipelines.
More from the Site Reliability and Outage Management trail
- Lights Out, Systems On: Validating Instant Power Loss Readiness
- DrP: Meta’s Root Cause Analysis Platform at Scale
- Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and Optimization
- Diff Risk Score: AI-driven risk-aware software development
- How Meta keeps its AI hardware reliable