Migrating billions of records: moving our active DNS database while it’s in use

What this post added

This post details the technical challenges and solutions involved in migrating Cloudflare's active DNS database, containing billions of records, from a monolithic PostgreSQL cluster (cfdb) to a new, partitioned cluster (dnsdb). Key contributions include the design and implementation of a custom Change Data Capture and Transfer Service (CDCTS) to ensure near real-time data synchronization and minimize downtime, the logical decoupling of DNS record data from zone data, and the introduction of a gRPC API for controlled access to DNS data. The migration also involved partitioning the primary DNS record table (_cf_rec_) based on zone_id and implementing bidirectional data transfer capabilities for rollback.