Supporting large campaigns at scale

This post details the development of a batch job service, part of Grab's real-time automation engine (Trident), to support large-scale marketing campaigns. It addresses the need for quick execution, high turnover rate, and maximized resource utilization by moving from a naive single-threaded approach to a distributed system using Kafka. Key technical contributions include: 1. Using Kafka for distributing batches of users (approx. 100 users per batch) to multiple servers for parallel processing. 2. Implementing partition keys for Kafka to ensure even distribution of batches. 3. Developing batch endpoints in downstream services (Rewards, Messaging) to process multiple users and actions in a single API call, significantly reducing QPS and network latency. 4. Optimizing database queries within batch endpoints by grouping users for batch updates. 5. Further optimizations include sharding Kafka streams by action type and country to enable parallel processing of different campaign types and geographical regions. 6. Making messaging actions asynchronous to avoid blocking subsequent user batch processing.