The journey of building a comprehensive attribution platform

This post details the evolution of Grab's attribution platform from limited capabilities to a comprehensive solution using Kappa and Lambda architectures. It describes the technical implementation of an Ads Attribution Engine using Kafka, DDB, S3, an in-house stream processing framework, Redis, ScyllaDB, and RDS. It then details the evolution to a unified engine incorporating promo touchpoints, using Druid for analytics. Finally, it outlines the transition to a full-fledged Lambda architecture with Coban for real-time processing and Spark-based ETL for batch processing, leveraging S3 as a data warehouse. The post also discusses challenges with Lambda architecture and proposes an optimization strategy to separate real-time and batch pipelines.