Enabling near real-time data analytics on the data lake

This post details Grab's implementation of Hudi for near real-time data analytics on its data lake. It describes configurations for high/low throughput and unbounded/bounded data sources (Kafka/RDS). For high throughput, MOR format with Flink and Spark is used. For low throughput, COW format with Flink is used. It addresses Protobuf to Hudi schema transformation for Kafka sources and Flink CDC for RDS sources. It also discusses indexing strategies (Bucket Index vs. Flink State Index) and the choice of Bucket Index for unbounded sources, with future exploration of consistent hashing. The impact on business metrics and fraud detection is highlighted, along with future plans to integrate Record Level Indexing and explore other table formats like Iceberg and Delta Lake.