Preparing for the worst: Our core database failover test
8/28/2025
This post details the successful execution of a full production failover of Vercel's core control-plane database from Azure West US to East US 2. It highlights the motivation for rigorous disaster recovery testing, the architectural design for resilience across 19 regions, and the specific engineering challenges encountered and resolved during preparation. These challenges included making proprietary Cosmos DB clients failover-aware by programmatically updating write targets at runtime, reducing write region detection delays from five minutes to five milliseconds, and addressing partition-level inconsistencies during failover. The post also describes the live exercise, its operational impact, and the validation of the on-call team's readiness through alert and runbook integration. Lessons learned from both graceful and staged offline failovers are being used to inform the product roadmap.