Under the Hood: Automated backups

This post details the implementation of a three-stage automated backup system for Facebook's large-scale MySQL installations. Stage 1 involves rack backups (RBUs) collecting binary logs and using mysqldump for nightly logical backups. Stage 2 utilizes Hadoop DFS for stable, replicated storage of these backups. Stage 3 involves weekly copying to discrete storage in a separate region. The post also describes a novel exponential scoring system for backup failure severity monitoring and a continuous restore testing process with a self-service restore coordinator (ORC).