BellJar: A new framework for testing system recoverability at scale

What this post added

Introduced BellJar, a new framework for testing system recoverability at scale. BellJar creates isolated, vacuum-sealed environments that mimic production during total infrastructure outages, allowing engineers to test recovery strategies and uncover dependencies. It uses virtualization and fault injection to simulate various failure modes and allows granular allowlisting of production capabilities needed for recovery. The framework enables programmatic definition of recovery scenarios, including the service under test, hardware, recovery strategy, validation criteria, tooling, and recovery conditions, facilitating repeatable testing and integration into CI/CD pipelines.