Messaging System Redesign

5 posts · Feb 2018 – Dec 2024

What this trail covers

Grab's real-time data platform (Coban) has evolved its data integration capabilities by adopting Kafka Connect for seamless data movement into and out of Apache Kafka. This enables Change Data Capture (CDC) from databases like MySQL and replication to disaster recovery sites, supporting various data sources and sinks. The system leverages Debezium for MySQL/Aurora binlog processing and Lambda functions for DynamoDB stream processing, ensuring data consistency and reducing load on source databases. This post details a custom time-based switchover logic for migrating stream processing functionality to a new service with zero data loss and duplication, handling high volumes of real-time traffic.

All Grab trails

Dec 2024How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplicationIntroduced a custom time-based switchover logic implemented in shared code between existing and new services to seamlessly migrate high-volume real-time streaming traffic from one service to another. This approach ensured zero data loss and duplication by gradually migrating streams one by one, validating the new service's processing in production before full switchover, and performing the switchover with no downtime or data inconsistencies. The shared code allowed both services to import and use the same stream processing logic, with selective turning on/off based on the service and scheduled switchover times. Temporary 'sink' resources were created for validation. The migration was completed over three weeks.
Apr 2023Message Center - Redesigning the messaging experience on the Grab superappThis post details the redesign of Grab's messaging experience into 'Message Center', replacing GrabChat. It outlines the new system's architecture, comprising a message processor backend and a scalable message delivery service ('postman'). Key technical decisions include using Apache Kafka for event streaming, Amazon SQS for retries, and TCP via Hermes for low-latency, bi-directional client-server communication. The implementation of an ACK system to ensure message delivery and handle dropped connections is highlighted, along with the use of Hedwig for push notifications. The post emphasizes decoupling services, understanding requirements for extensibility, and implementing safeguards against system failures.
Apr 2022How Kafka Connect helps move data seamlesslyThis post details Grab's adoption and implementation of Kafka Connect to solve data integration challenges. It covers the Kafka Connect stack in Grab, including the use of Protobuf and Confluent Schema Registry. It addresses problems like Change Data Capture (CDC) by leveraging Debezium connectors for MySQL, enabling atomic writes and providing data snapshots. It also discusses message mirroring for disaster recovery using MirrorMaker2 (MM2) deployed on Kafka Connect for finer control and infrastructure-as-code management, and for cross-cloud ingestion to Azure Event Hubs with a custom Protobuf-to-JSON converter. The deployment strategy using Docker containers on Kubernetes and Terraform for connector management is outlined. Future plans include a unified control plane for Kafka resources and connectors.
Mar 2022Real-time data ingestion in GrabThis post details Grab's implementation of a real-time data ingestion system to synchronize data from MySQL, Aurora, and DynamoDB to Kafka. It highlights the challenges of dual writes and data integrity issues with previous approaches. The architecture includes stream storage (binlog/DynamoDB streams), event producers (Debezium for MySQL/Aurora, Lambda for DynamoDB), Kafka as the message queue, and a Golang library as a stream processor. Use cases include powering data pipelines with less database pressure, enabling business decisions via event-driven architectures (e.g., Grab Search, GrabFood), incremental database replication for disaster recovery, and delivering audit trails.
Sep 2021Search indexing optimisationThis post details optimizations to the incremental sync process for Elasticsearch indexing by leveraging MySQL binlogs. The original system suffered from heavy DB load, data loss, tight coupling with MySQL table structure, redundant Elasticsearch updates, and duplicate cascade updates. The optimized approach uses MySQL row-based binlogs captured by a CDC system. Stream events now have a common structure with Operation, PayloadBefore, and PayloadAfter. Optimizations include filtering irrelevant stream events by comparing PayloadBefore and PayloadAfter against the Elasticsearch data subset, and changing to partial updates using data differences from the Binlog event. This resulted in no data loss, no dependency on MySQL table definition, no redundant Elasticsearch updates and DB reads, a 90% reduction in Elasticsearch reads traffic, filtering out 55% of irrelevant stream events, and a 55% reduction in DB load. Further optimizations changed most Elasticsearch relevant events to partial updates, leading to only changed fields being sent to Elasticsearch and an 80% reduction in DB load based on Optimisation 1.
Advertisement
Jul 2021Reshaping Chat Support for Our UsersThis post details the product and design approach to building an in-house chat tool for support, focusing on preventing disconnections, improving agent productivity, integrating with internal systems, and routing users to specialists. It covers the MVP experiment, scaling efforts including reducing waiting time and time to reply with ML-driven suggestions and dynamic queue limits, and overall chat time reduction through visibility prompts. Key learnings emphasize starting with the end in mind, leveraging data, remote user testing, addressing root causes, and shadowing users.
Jun 2020How We Built Our In-house Chat Platform for the WebThis post details the technical implementation of an in-house chat platform for the web, focusing on the challenges of handling multiple browser tabs and maintaining a single TCP connection. It introduces the use of JavaScript's SharedWorker and BroadcastChannel APIs to manage state synchronization across tabs and abstract the underlying transport mechanism. The architecture involves a TCP gateway for mobile apps and a hybrid approach for the web using SharedWorker to manage connections and BroadcastChannel for inter-tab communication, ensuring a seamless user experience.
Jan 2020Plumbing At ScaleThis post details the journey of building and deploying an event sourcing platform in Go, building a stream processing framework over it, and scaling it to service over 300 billion events a week. It covers the core concepts of event sourcing, the case for stream processing, platform requirements (scaling, NoOps, multi-tenancy, resiliency, tunable tradeoffs), the capabilities of the Go stream processing framework (deduplication, filtering, mapping, aggregation, windowing, joins, chaining, rewind, replay, sinks, error handling), and the architecture of the Coban platform, including its deployment infrastructure on Kubernetes and the metastore using ScyllaDB.
May 2019Tourists on GrabChat!This post analyzes tourist usage patterns of GrabChat, focusing on the utility of its auto-translation and template features. It provides data on the volume of messages, translation usage, and template adoption by tourists across different countries. It also highlights the use of image sharing for location-based communication in complex pickup areas. The analysis correlates template usage with higher ride completion rates and identifies common communication themes related to luggage, children, and pets.
Mar 2018How Grab Experimented with Chat to Drive Down Booking CancellationsThis post details experiments conducted on GrabChat to reduce booking cancellations. It describes how automated messages, sent at different delay intervals and with varying verbiages and tones, were tested. The experiments revealed that earlier and more direct/personalized messages were more effective in reducing cancellations. The key learning was that the quality of engagement, not just the quantity of messages, was the primary driver for reducing cancellations. This contributed to a reduction in booking cancellations by up to 2 percentage points.