TechPrufer
Sign inGet started

Blogs›Uber›Data Warehouse Decentralization and Federation

Data Warehouse Decentralization and Federation

Data Warehouse Decentralization and Federation

3
posts
2024–2026

Uber is undertaking a significant initiative to decentralize its monolithic data warehouse, addressing challenges of shared-fate outages, resource contention, operational bottlenecks, governance blind spots, and open ACLs. This evolution involves migrating large datasets into smaller, domain-specific units, leveraging a 'pointer manipulation' technique for zero-downtime migrations without data duplication. The system comprises a layered access model for security, with foundational primitives managed by a Bootstrap Migrator, Realtime Synchronizer, Batch Synchronizer, and Recovery Orchestrator. The Bootstrap Migrator handles one-time data copying and pointer updates, ensuring data integrity through distributed copying, checksum verification, and rigorous data quality checks. The pointer manipulation strategy updates the Hive Metastore to point to new HDFS locations, enabling seamless transitions and significant cost savings by avoiding data duplication.

2026

Simplifying Data and Product Integrations with a Data Abstraction Layer

6/11/2026

This post introduces the Data Abstraction Layer (DAL) as a key component in simplifying data access and decoupling data consumers from producers. It details the DAL's architecture, request/response flow, and the underlying table resolution process. The DAL addresses the challenges of evolving data models and heterogeneous data sources by providing a unified interface for accessing various data types, exemplified by its application in advertiser reporting. This contributes to the broader goal of managing and accessing data more effectively within Uber's decentralized data ecosystem.

Database Federation: Decentralized and ACL-Compliant Hive™ Databases

2/19/2026

This post details the migration strategy for decentralizing Uber's monolithic Hive data warehouse. It introduces the 'pointer manipulation' technique, where Hive Metastore entries are updated to point to new HDFS locations, enabling zero-downtime migrations without data duplication. The post outlines the four main components of the migration system: Bootstrap Migrator, Realtime Synchronizer, Batch Synchronizer, and Recovery Orchestrator. It elaborates on the Bootstrap Migrator's functionalities, including migration configuration, pipeline management, pre-migration checks, backup pointers, distributed data copying with checksum verification, target dataset location and partition recovery, and data quality checks. The core advantage highlighted is the cost savings and simplified data pipelines achieved by avoiding data duplication.

2024

Enabling Security for Hadoop Data Lake on Google Cloud Storage

7/30/2024

This post details the technical challenges and solutions for migrating Uber's on-premise Hadoop data lake to Google Cloud Storage (GCS) on GCP. It focuses on the security aspects, specifically addressing the integration of Hadoop's Kerberos-based authentication and authorization model with GCS's IAM. Key contributions include the design of a layered access model (Foundations, Data Mesh, Dataset Access) for enhanced security in the cloud, and the development of a Storage Access Service as an intermediary to manage authentication and authorization, overcoming limitations of native GCP IAM for large-scale policy management and reducing attack vectors. The post also elaborates on the existing HDFS security architecture and the authentication flow to highlight the differences and integration points.