TechPrufer
Get started
Uber
Uber logoUber · Feature trail

Data Warehouse Decentralization and Federation

5 posts · Jul 2024 – Aug 2026

What this trail covers

Uber has developed a chunked aggregation buffer strategy to overcome JVM memory limitations when computing exact distinct counts for high-cardinality non-rollup metrics in distributed data pipelines. This strategy partitions the bitmap aggregation buffer across a map, keyed on the top 16 bits of each hashed identifier, effectively eliminating the 2 GB JVM constraint and enabling concurrent execution of backfill jobs. This has reduced two-year backfill time by 65% on average and eliminated out-of-memory failures. The solution is generalizable beyond Uber's stack and addresses challenges in computing metrics like monthly active users at scale.

All Uber trails
  1. Running Cost-Efficient Export Workloads at UberThis post details how Apache Hudi column stats in conjunction with sorting/clustering can reduce scan volume and make export workloads more cost-efficient in a Google Cloud Storage-backed lakehouse environment. It addresses the problem of export workloads (e.g., DSAR requests) causing full-table scans, which disrupts GCS auto-class tiering by keeping data hot and increasing storage, retrieval, and metadata operation costs. The solution combines Hudi column stats (min, max, null count) stored in the metadata table to prune files without touching data, and table sorting on predicate columns to cluster matching records into fewer files, making pruning more effective. Benchmarks show significant reductions in files touched, GCS egress, compute, and storage.
  2. Scaling Exact COUNT(DISTINCT) for High-Cardinality Non-Rollup Metrics in Distributed Data PipelinesIntroduced a chunked aggregation buffer strategy for computing exact distinct counts of high-cardinality non-rollup metrics in JVM-based data pipelines. This strategy partitions the aggregation buffer using the top 16 bits of hashed identifiers, creating independent chunks that serialize and deserialize separately. This circumvents the JVM's 2 GB array indexing limit, allowing for safe computation of cardinalities up to approximately 11.7 trillion. The solution also enables concurrent execution of backfill jobs by eliminating shared state and ensures backward compatibility with legacy monolithic bitmap formats through a self-describing magic number router. The final output is a cardinality-only 8-byte long, avoiding bitmap serialization at the reduce stage.
  3. Simplifying Data and Product Integrations with a Data Abstraction LayerThis post introduces the Data Abstraction Layer (DAL) as a key component in simplifying data access and decoupling data consumers from producers. It details the DAL's architecture, request/response flow, and the underlying table resolution process. The DAL addresses the challenges of evolving data models and heterogeneous data sources by providing a unified interface for accessing various data types, exemplified by its application in advertiser reporting. This contributes to the broader goal of managing and accessing data more effectively within Uber's decentralized data ecosystem.
  4. Database Federation: Decentralized and ACL-Compliant Hive™ DatabasesThis post details the migration strategy for decentralizing Uber's monolithic Hive data warehouse. It introduces the 'pointer manipulation' technique, where Hive Metastore entries are updated to point to new HDFS locations after a one-time data copy, enabling zero-downtime migrations without data duplication. It also describes the architecture of the migration system, including the Bootstrap Migrator, Realtime Synchronizer, Batch Synchronizer, and Recovery Orchestrator, and elaborates on the Bootstrap Migrator's components: Migration Config, Migration Pipeline Driver, Pre-Migration Checks, Backup Pointers, Distributed Data Copier, Checksum Verifier, Target Dataset Location and Partition Recovery, Data Quality checks, and Source Dataset Pointer Update.
  5. Enabling Security for Hadoop Data Lake on Google Cloud StorageThis post details the security architecture for migrating Uber's on-premise Hadoop data lake to GCP, specifically addressing the challenges of integrating the existing Hadoop security model (Kerberos, Delegation Tokens, HDFS ACLs) with Google Cloud Storage (GCS) and GCP IAM. It introduces a layered access model (Foundations, Data Mesh, Dataset Access) and a custom Storage Access Service to manage authentication and authorization, overcoming GCS IAM limitations and reducing attack vectors. The system bridges HDFS and GCS security models to enable secure data lake migration.