TechPrufer
Sign inGet started

Blogs›Grab›Data Lake Modernization with Apache Iceberg

Data Lake Modernization with Apache Iceberg

Data Lake Modernization with Apache Iceberg

2
posts
2026

Grab has adopted Apache Iceberg as the foundational table format for its data lake, migrating from Hive Parquet. This transition addresses challenges with catalog latency, the small file problem, operational toil, and data consistency. The adoption has led to significant improvements in query performance (e.g., 10x reduction in runtime via Z-ordering), reduced S3 API costs (up to 95%), and compute savings. A key component of this migration is the open-sourced UnifiedSparkCatalog, which provides a unified interface for data lake operations. The Hugo platform has been re-engineered with Apache Flink to provide a unified, one-click data ingestion platform, supporting both MySQL CDC and Kafka streams directly into the data lake, significantly reducing onboarding time and operational overhead.

2026

Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption

7/24/2026

This post details Grab's migration from Hive Parquet to Apache Iceberg for its data lake. It outlines the challenges faced with the previous directory-based Hive Metastore approach, including catalog latency, the small file problem, and operational overhead. The post explains the strategic decision to adopt Iceberg, highlighting its benefits for consistency and performance. It showcases the substantial efficiency gains achieved through Iceberg adoption, such as 10x query performance improvement with Z-ordering and significant S3 API cost reductions. A major contribution is the introduction and open-sourcing of the UnifiedSparkCatalog, a Spark catalog that abstracts away table format differences, enabling seamless coexistence and migration of tables across Iceberg, Delta, Hudi, and Hive. The post also discusses lessons learned, including handling Hive lock contention and timestamp compatibility issues, and outlines future directions like Storage Partitioned Joins and Apache XTable.

The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache Flink

7/24/2026

This post details the evolution of Grab's Hugo data ingestion platform, migrating from a Spark-based batch processing system to a unified, self-service platform powered by Apache Flink. It describes the architectural shift from siloed workflows using Kafka Connect and Sprinkler to a streamlined Flink-based approach for both MySQL CDC and Kafka ingestion. Key technical contributions include the elimination of intermediary Kafka hops for CDC, automated schema detection and dynamic validation for Kafka streams, and a reduction in pipeline components from four to two. The post highlights significant improvements in onboarding time (from days to minutes) and increased adoption rates.