Code Orange: Fail Small — our resilience plan following recent incidents
Dec 19, 2025Dane Knecht

Summary
This post details the 'Code Orange: Fail Small' initiative, a prioritized plan to enhance network resilience following recent incidents. Key contributions include: 1. Implementing controlled rollouts for configuration changes, mirroring software release processes, using an enhanced Quicksilver system. 2. Reviewing and improving failure mode handling between critical services by assuming failure and implementing sane defaults. 3. Revising 'break glass' procedures and removing circular dependencies to expedite emergency response.