Continuous MySQL backup validation: Restoring backups

This post details the ORC (ORC Restore Coordinator) system, a component of Meta's continuous MySQL backup validation infrastructure. It describes the architecture of ORC, including the Warchief (load balancer), ORC DB (state management), and Peons (restore workers). The post elaborates on the five stages of a restore job within a peon: SELECT, DOWNLOAD, LOAD, VERIFY, and REPLAY. A significant technical contribution is the detailed explanation of the binlog selection logic, which leverages Global Transaction Identifiers (GTIDs) to accurately determine and replay transactions for point-in-time restores, overcoming challenges with primary/secondary backup sources and ensuring data integrity.