Database Failover and Resilience Testing

Vercel's platform is designed for resilience, with each of its 19 cloud regions operating autonomously. In the event of a regional outage, the anycast routing layer removes the affected region from rotation within seconds, automatically rerouting traffic with no disruption. Compute is also resilient, with backup regions for function execution. The platform has undergone rigorous testing, including full production failovers of its core control-plane database, to ensure minimal disruption and zero data loss. This post details the migration of the build warm pool's state from Redis to DynamoDB, a critical system that tracks container readiness, authentication tokens, and billing mappings. The migration was performed live, in phases, using feature flags and shadow modes to ensure data durability and consistency. Key challenges included handling the continuous nature of the build pool, the latency differences between Redis and DynamoDB, and the need to redesign the supply loop to accommodate slower reads. The migration involved designing a new DynamoDB schema based on access patterns, implementing a phased rollout with extensive monitoring and validation, and ultimately rebuilding the supply loop to run concurrently around the new latency characteristics.