Real-time data ingestion in Grab
Mar 14, 2022Shuguang Xiang
Summary
This post details the implementation of a real-time data ingestion system at Grab, focusing on synchronizing data from MySQL, Aurora, and DynamoDB to Kafka. It covers the architecture, including stream storage (binlogs and DynamoDB streams), event producers (Debezium, Lambda), Kafka, and a Golang stream processor. The post highlights the benefits for data pipelines, business decisions, database replication, and audit trails, emphasizing improved data consistency and reduced database read pressure.
More from the Data Production Issue (DPI) Operationalization trail
- Data Mesh at Grab (Part III): Operationalizing data reliability with automated DPIs
- The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache Flink
- Data Mesh at Grab (Part II): The foundational tools behind certification
- Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test
- Data mesh at Grab part I: Building trust through certification