TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Fare Storage and Event Sourcing

1 post · Nov 2020 – Aug 2022

What this trail covers

Grab's Order Platform has evolved its database solution to handle millions of orders daily by separating transactional (OLTP) and analytical (OLAP) workloads. DynamoDB is used for OLTP with a focus on stability, scalability, and cost-effectiveness, employing strategies like Global Secondary Indexes (GSIs) with sparse indexing for ongoing orders and Time To Live (TTL) for data retention. MySQL RDS serves as the OLAP database for historical and statistical queries, utilizing partitioning for data. The Fare Storage system has been refactored to use Event Sourcing and CQRS patterns, with DynamoDB as the event store and a cache for reads. Optimistic locking and transaction IDs are used to ensure data integrity and idempotency. A schema-less metadata approach via an SDK decouples clients from the storage service.

All Grab trails
  1. How we store and process millions of orders dailyThis post details the design and implementation of a dual-database architecture for Grab's Order Platform to handle millions of daily orders. It introduces DynamoDB for OLTP with specific strategies for primary key and range queries using GSIs and sparse indexes, and a phased approach to TTL for data retention. For OLAP, MySQL RDS is used with monthly partitioning. A Kafka-based data ingestion pipeline is described for eventual consistency, including mechanisms for handling duplicate and out-of-order messages via database unique keys, version updates, and upserts. The solution emphasizes stability, scalability, cost-effectiveness, and consistency, resulting in significant cloud cost savings.
  2. Democratising Fare Storage at Scale Using Event SourcingThis post details the implementation of Event Sourcing and CQRS patterns for Grab's Fare Storage. It introduces DynamoDB as the event store, backed by a cache for reads, and discusses the use of optimistic locking with fare versions and unique transaction IDs to ensure data integrity and idempotency during concurrent updates. A schema-less metadata approach using an SDK is also described, allowing clients to evolve metadata without modifying the storage service.