Migrating billions of records: moving our active DNS database while it’s in use

This post details the migration of Cloudflare's active DNS database, containing billions of records, from the main cfdb cluster to a new dnsdb cluster. The migration was driven by storage strain on cfdb and the need for better control over DNS data access. Key technical challenges and solutions included: decoupling zone data from DNS record data, creating a gRPC API for DNS records to centralize access and enable rate limiting/caching, migrating over 20 services to use this API, and moving auditing and zone setting logic to the application level. A Change Data Capture and Transfer Service (CDCTS) was built to handle the initial data copy and near real-time change replication, prioritizing data integrity and minimal downtime. The CDCTS design considered bidirectional migration capabilities, partitioning of the _cf_rec table by zone_id, and the use of foreign tables for emergency access. The _cf_rec table contained 1.7 billion records and 1.5 TB of data, with millions of daily inserts, updates, and deletes. The _cf_archived_rec table contained 4.3 billion records. The migration process involved an initial copy and a continuous change copy mechanism.