Running Cost-Efficient Export Workloads at Uber

This post details how Apache Hudi column stats in conjunction with sorting/clustering can reduce scan volume and make export workloads more cost-efficient in a Google Cloud Storage-backed lakehouse environment. It addresses the problem of export workloads (e.g., DSAR requests) causing full-table scans, which disrupts GCS auto-class tiering by keeping data hot and increasing storage, retrieval, and metadata operation costs. The solution combines Hudi column stats (min, max, null count) stored in the metadata table to prune files without touching data, and table sorting on predicate columns to cluster matching records into fewer files, making pruning more effective. Benchmarks show significant reductions in files touched, GCS egress, compute, and storage.