How Kafka Connect helps move data seamlessly

This post details Grab's adoption and implementation of Kafka Connect for various data integration scenarios. It covers the use of Debezium connectors for CDC from MySQL, leveraging binlog streams for data changes and snapshots, and handling DB upgrades/outages by managing connector offsets. It also explains the deployment of MirrorMaker2 on Kafka Connect for message mirroring and offset translation, enabling disaster recovery and zero-downtime migrations. Furthermore, it describes the development of a custom Protobuf-to-JSON converter for ingesting data into Azure Event Hubs. The deployment architecture using Docker and Kubernetes, managed by Terraform, is also outlined.