Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption

This post details the migration from Hive Parquet to Apache Iceberg for Grab's data lake. It highlights the challenges with Hive Parquet, including catalog latency, the small file problem, operational toil, and broken information loops. The adoption of Iceberg is justified by its community governance, engine compatibility, and long-term flexibility, leading to significant improvements in query performance (e.g., 10x improvement with Z-ordering), S3 API cost reduction (up to 95%), and compute savings. A key contribution is the open-sourcing of the UnifiedSparkCatalog, a Spark catalog that abstracts table format differences (Iceberg, Delta, Hudi, Hive) for users, ensuring transparency and simplifying migrations. The post also discusses lessons learned, such as managing Hive lock contention and timestamp handling differences between Spark and Iceberg.