Database Federation: Decentralized and ACL-Compliant Hive™ Databases

This post details the migration strategy for decentralizing Uber's monolithic Hive data warehouse. It introduces the 'pointer manipulation' technique, where Hive Metastore entries are updated to point to new HDFS locations after a one-time data copy, enabling zero-downtime migrations without data duplication. It also describes the architecture of the migration system, including the Bootstrap Migrator, Realtime Synchronizer, Batch Synchronizer, and Recovery Orchestrator, and elaborates on the Bootstrap Migrator's components: Migration Config, Migration Pipeline Driver, Pre-Migration Checks, Backup Pointers, Distributed Data Copier, Checksum Verifier, Target Dataset Location and Partition Recovery, Data Quality checks, and Source Dataset Pointer Update.