Data Lake Modernization with Apache Iceberg

6 posts · May 2017 – Jul 2026

What this trail covers

Grab has adopted Apache Iceberg as the foundational table format for its data lake, migrating from Hive Parquet. This transition addresses challenges with catalog latency, the small file problem, operational toil, and data consistency. A key component of this migration is the open-sourced UnifiedSparkCatalog, which provides significant improvements in query performance, reduced S3 API costs, and compute savings. The Talaria time-series database has evolved to use Go plugins for custom data processing.

All Grab trails

Jul 2026Scaling Grab's Data Lake: Our journey to Apache Iceberg adoptionThis post details the migration from Hive Parquet to Apache Iceberg for Grab's data lake. It highlights the challenges with Hive Parquet, including catalog latency, the small file problem, operational toil, and broken information loops. The adoption of Iceberg is justified by its community governance, engine compatibility, and long-term flexibility, leading to significant improvements in query performance (e.g., 10x improvement with Z-ordering), S3 API cost reduction (up to 95%), and compute savings. A key contribution is the open-sourcing of the UnifiedSparkCatalog, a Spark catalog that abstracts table format differences (Iceberg, Delta, Hudi, Hive) for users, ensuring transparency and simplifying migrations. The post also discusses lessons learned, such as managing Hive lock contention and timestamp handling differences between Spark and Iceberg.
Jul 2026The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache FlinkThe Hugo platform has been engineered with a custom automation layer and Apache Flink to unify data ingestion workflows. This evolution replaces siloed onboarding processes with a seamless, one-click experience for both MySQL CDC and Kafka ingestion. For MySQL CDC, Flink directly reads the database binlog, eliminating intermediary components like Kafka Connect and Sprinkler, reducing operational overhead from 4 to 2 core components, automating schema detection, and enabling instant onboarding. For Kafka ingestion, Flink dynamically retrieves Protobuf schemas from Confluent Schema Registry, removing hardcoding and manual stream registration, and enabling click-to-query functionality in the Hugo UI. This has reduced onboarding time for Kafka pipelines to approximately 6 minutes and for MySQL CDC pipelines to approximately 3 minutes, leading to a significant increase in pipeline adoption.
Feb 2024Enabling near real-time data analytics on the data lakeThis post details Grab's implementation of Hudi for near real-time data analytics on its data lake. It describes configurations for high/low throughput and unbounded/bounded data sources (Kafka/RDS). For high throughput, MOR format with Flink and Spark is used. For low throughput, COW format with Flink is used. It addresses Protobuf to Hudi schema transformation for Kafka sources and Flink CDC for RDS sources. It also discusses indexing strategies (Bucket Index vs. Flink State Index) and the choice of Bucket Index for unbounded sources, with future exploration of consistent hashing. The impact on business metrics and fraud detection is highlighted, along with future plans to integrate Record Level Indexing and explore other table formats like Iceberg and Delta Lake.
Jan 2024Rethinking Stream Processing: Data ExplorationThis post introduces the adoption of Apache Zeppelin as a web-based notebook environment to enable data exploration on online data (streaming data from Kafka). It details the problem of underutilized streaming data due to exploration difficulties, the solution of using Zeppelin with Flink interpreters for SQL-based querying and visualization, and the need for dynamic schema derivation from Protobuf to DDL. The post also touches upon security requirements for data access authorization and audit trails.
May 20232.3x faster using the Go plugin to replace Lua virtual machineThis post details the migration of Talaria's data ingestion processing from Lua VMs to Go plugins. The Lua VM approach suffered from significant performance and memory overhead due to launching a new VM for each script. By leveraging Go plugins (compiled as shared libraries), Talaria achieved a 2.3x speedup and a 2.3x reduction in memory usage for custom data processing functions, bringing performance closer to native Go functions. The implementation involved creating a `plugin` loader that mirrors the `Handler` interface of the `LuaLoader`, allowing for dynamic loading and execution of Go functions from shared libraries.
Advertisement
Aug 2019Data First, SLA AlwaysIntroduced Trailblazer, a Spark Structured Streaming application for Change Data Capture (CDC) from MySQL databases using Debezium and Kafka. This addresses the limitations of periodic batch ingestion and improves scalability for large tables by streaming binlogs in real-time to the data lake. Key design principles for Trailblazer include externally managed checkpoints (using Redis), handling varying data volumes through Spark configurations, using data as a health indicator, and treating streams as ephemeral. Monitoring strategies for these streams include tracking the running:active jobs ratio, microbatch runtime, and job liveliness.
Jan 2019Querying Big Data in Real-time with Presto & Grab's TalariaDBThis post details the design and implementation of TalariaDB, a distributed, highly available, and low-latency time-series database for storing real-time data like logs, metrics, and click streams. TalariaDB addresses the need to query terabytes of data per hour with predictable low latency and cost, storing data in pre-partitioned batches in a binary, columnar format. It leverages the Badger key-value store and integrates with Presto via the Presto Thrift Connector for SQL querying. The system focuses on simplicity, linear horizontal scalability, low latency, availability, zero operation, high write throughput, and cost-effectiveness.
May 2017Scaling Like a Boss with PrestoThis post details Grab's transition from Amazon Redshift to a data lake architecture using Amazon S3 and Presto for analytics. Faced with scaling challenges due to increasing user numbers and peaky workloads on Redshift, Grab built a data lake using S3, storing data in AVRO format partitioned by timestamp. Spark was used for data pipelines and transformations. To address the limitations of Spark and Zeppelin for SQL-based users, Presto was introduced as the SQL engine over the data lake. The post discusses the challenges of AVRO format incompatibility with Presto, leading to a switch to Parquet. Performance benchmarks comparing Presto against Redshift are presented, highlighting Presto's strengths in partitioned data queries and its ability to support a shared-data multi-cluster architecture. The migration of SQL workloads from Redshift to Presto involved building custom UDFs and learning optimized query writing for Presto.