Cloudflare

Code Orange: Fail Small — our resilience plan following recent incidents

Dec 19, 2025Dane Knecht
Code Orange: Fail Small — our resilience plan following recent incidents
SummaryAI, from the source post

This post details the 'Code Orange: Fail Small' initiative, a prioritized plan to enhance network resilience following recent incidents. Key contributions include: 1. Implementing controlled rollouts for configuration changes via the Quicksilver system, mirroring existing Health Mediated Deployment (HMD) for software releases. 2. Reviewing and improving interface contracts between critical services to ensure graceful failure handling, including using validated defaults and allowing traffic to pass with passable classifications when modules fail. 3. Revising 'break glass' procedures and removing circular dependencies to enable faster access to tools during emergencies while maintaining security.

More from the Process Management & Graceful Upgrades trail