Migrating Data Ingestion Systems at Meta Scale

This post details the successful large-scale migration of Meta's data ingestion system from a legacy architecture to a new, self-managed data warehouse service. Key contributions include the establishment of a three-phase migration lifecycle (Shadow Phase, Reverse Shadow Phase, Migration Cleanup) with defined success criteria (data quality, landing latency, resource utilization). It also describes the development of a custom data quality analysis tool using Scuba for real-time comparison of row counts and checksums, and strategies for handling rollout and rollback, including early signals via backfills and mechanisms to stop bad data propagation in CDC processes. The post highlights the use of automated tooling for monitoring and migrating tens of thousands of jobs in batches, considering factors like throughput and priority, and managing limited migration capacity.