TechPrufer
Get started
Dropbox
Dropbox logoDropbox · Feature trail

Disaster Readiness and Data Center Failover

1 post · Apr 2022

What this trail covers

Dropbox has significantly improved its disaster readiness by reducing its Recovery Time Objective (RTO) through a year-long effort culminating in the physical unplugging of its San Jose data center. This involved transitioning from an active-passive to a more robust active-active architecture for its metadata stack, addressing complexities in MySQL replication, consistency, and data modeling. The creation of a dedicated Disaster Readiness (DR) team and the development of modular, safe failover tooling were key to achieving this.

All Dropbox trails
  1. That time we unplugged a data center to test our disaster readinessThis post details the culmination of over a year's work by the Disaster Readiness (DR) team to physically unplug the San Jose data center and test disaster preparedness. It highlights the architectural evolution from a centralized San Jose data center to a multi-homed, active-active Magic Pocket for block storage, and the challenges in achieving an active-active metadata stack due to MySQL replication lag, read-committed isolation, and complex data ownership in Edgestore. The post also describes the creation of the dedicated DR team and the development of modular, safe failover tooling, inspired by Facebook's Maelstrom paper, to address issues identified during a previous outage.