Upgrading MySQL at Shopify - Shopify

This post details the process of upgrading Shopify's MySQL instances from Percona Server 5.7.21 to 5.7.32. It describes the development of the Declarative Database Topologies project, an extension of the Mason self-healing service, to automate the upgrade of thousands of VM-based database shards. The post highlights the discovery and resolution of a critical rollback issue where downgrading from 5.7.32 to 5.7.21 caused unexpected table rebuilds due to changes in `innodb_index_stats` and `innodb_table_stats` column lengths and the interaction of `ALTER TABLE` commands with `mysql_upgrade`. This involved patching Percona Server's `mysql_system_tables_fix.sql` and debugging the MySQL server source code to identify the root cause of the incorrect index change detection.