Preparing for the worst: Our core database failover test

This post details Vercel's successful execution of a full production failover of its core control-plane database from Azure West US to East US 2. The exercise involved all control-plane traffic, including API requests, background jobs, and deployment/build operations, with zero customer impact. Key technical challenges addressed included proprietary Cosmos DB clients lacking failover awareness, which were patched to detect and redirect writes at runtime with a reduced delay from five minutes to five milliseconds. The post also discusses workarounds for elevated latencies and stalled writes caused by disagreements on write regions during the transition, and the successful resolution of these issues across 57 staging failovers. The live exercise resulted in minor operational impact, primarily to Vercel's own builds, and validated the resilience of the platform and the readiness of on-call teams through targeted and catch-all alerts.