Blogs›Shopify›MySQL Backup and Restore Infrastructure

MySQL Backup and Restore Infrastructure

posts

2019

Shopify has redesigned its MySQL backup and restore tooling to leverage GCP's Persistent Disk (PD) snapshot feature. This has significantly reduced the Recovery Time Objective (RTO) to under 30 minutes. The new system uses Kubernetes CronJobs for frequent, automated PD snapshots across all regions, minimizing data loss and enabling faster replica rebuilds. A robust retention policy manages snapshot storage costs. A verification process ensures data integrity by restoring and checking daily snapshots for instance startup, replication, and corruption. For offsite disaster recovery, data from snapshots is compressed, encrypted, and archived with a third-party provider. While PD snapshots offer speed and efficiency, they are more expensive for storage and require additional tooling for integrity checks and offsite archiving.

2019

How Shopify Manages Petabyte Scale MySQL Backup and Restore - Shopify

10/1/2019

This post details the redesign of Shopify's MySQL backup and restore solution, moving from Percona's Xtrabackup to GCP's Persistent Disk (PD) snapshot feature. Key technical contributions include: building tooling around the GCP API to invoke PD snapshots, deploying this tooling as Kubernetes CronJobs for automated, frequent (every 15 minutes) backups across regions, implementing a framework for snapshot retention policy management, developing a restore process that exports snapshots to new PD volumes, and establishing a backup verification process that exports daily snapshots to PD volumes and runs Kubernetes Jobs to check for instance startup, replication, and InnoDB page-level corruption. Additionally, tooling was created to export snapshots to an offsite location for disaster recovery.