How We Simplified Our Data Ingestion & Transformation Process

This post details the evolution of Grab's real-time data ingestion and transformation process. Initially, a system using Kafka for collection, Spark Streaming and a Golang service for transformation, and TalariaDB for storage was built. Challenges arose with operational complexity, Spark Streaming scaling issues, data loss due to S3 eventual consistency, and processing lag. The team simplified the architecture by re-implementing the transformation logic in Golang, addressing event partitioning, ORC encoding, and improving memory utilization. This resulted in a reduced processing lag to approximately 1 minute, fixed data loss issues, and decreased operational workload by removing Spark Streaming.