Code Orange: Fail Small — our resilience plan following recent incidents

What this post added

This post details the 'Code Orange: Fail Small' initiative, a prioritized plan to enhance network resilience following recent incidents. Key contributions include: 1. Implementing controlled rollouts for configuration changes via the Quicksilver system, mirroring existing Health Mediated Deployment (HMD) for software releases. 2. Reviewing and improving interface contracts between critical services to ensure graceful failure handling, including using validated defaults and allowing traffic to pass with passable classifications when modules fail. 3. Revising 'break glass' procedures and removing circular dependencies to enable faster access to tools during emergencies while maintaining security.