TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Data Production Issue (DPI) Operationalization

16 posts · Mar 2019 – Aug 2026

What this trail covers

Grab has evolved its data mesh operationalization by implementing an automated Data Production Issue (DPI) lifecycle. This system, orchestrated by Kinabalu, detects contract breaches via Genchi's data quality tests and automates triage, diagnosis (categorized into UPSTREAM_ERROR, PLATFORM_ERROR, JOB_ERROR, DATA_ERROR), and resolution. Hugo, the ingestion platform, plays a key role in diagnosing failures by consolidating signals and providing root cause analysis. Auto-resolution workflows handle common transient errors. The Data Health Workbench provides a centralized dashboard for monitoring and manual intervention. Future work includes advancing auto-resolution strategies, enriching the Data Health Workbench, and broadening diagnostics to include more infrastructure components like Flink.

All Grab trails
  1. Data Mesh at Grab (Part III): Operationalizing data reliability with automated DPIsThis post details the operationalization of Data Production Issues (DPIs) within Grab's data mesh. It introduces Kinabalu as the incident orchestrator that detects contract breaches via Genchi's data quality tests. The DPI lifecycle includes automated triage, diagnosis with a standardized error taxonomy (UPSTREAM_ERROR, PLATFORM_ERROR, JOB_ERROR, DATA_ERROR), and auto-resolution workflows. Hugo, the ingestion platform, is highlighted for its three-layered diagnosis architecture (signal collection, alert diagnosis, diagnosis result) that consolidates signals for root cause analysis and recommends pipeline remediations. The auto-resolution architecture automates recovery for transient failures and escalates complex issues. The post emphasizes the importance of diagnostic accuracy and consistent operational workflows for maintaining data reliability.
  2. The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache FlinkThis post details the evolution of Grab's Hugo data ingestion platform, migrating from a Spark-based batch system and fragmented streaming architecture (Kafka Connect, Sprinkler) to a unified, self-service platform powered by Apache Flink. Key contributions include: 1. Unified MySQL CDC pipelines using Flink CDC to directly read binlogs, eliminating intermediary components and manual schema mapping, reducing onboarding from days to minutes. 2. Automated Kafka ingestion pipelines that dynamically fetch Protobuf schemas from Schema Registry, simplifying schema evolution and enabling click-to-query functionality into Hive tables. 3. Retirement of Sprinkler and Kafka Connect for data lake ingestion, replaced by Flink and a custom automation layer. 4. Significant reduction in onboarding time for both Kafka (6 mins) and MySQL CDC (3 mins) pipelines, leading to a substantial increase in adoption.
  3. Data Mesh at Grab (Part II): The foundational tools behind certificationThis post details the foundational tools for data mesh certification at Grab: Hubble (metadata management and certification engine) and Genchi (data quality observability). Hubble, built on DataHub, provides search, ownership, data contracts, lineage, and computational certification status. Its event-driven architecture ensures metadata stays fresh. Genchi allows defining data quality tests (freshness, volume, schema, semantic validity) and integrates with pipeline completion events via Lighthouse's Sync with Pipeline (SWP) for event-driven test orchestration, moving away from cron-based schedules to ensure tests align with actual data production.
  4. Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic testIntroduced a real-time data quality monitoring solution for Kafka streams. This system allows users to define data contracts including schema agreements, semantic rules (string patterns, number ranges, constant values), and ownership details. It uses FlinkSQL to execute tests based on these contracts, identifying syntactic (schema mismatches) and semantic (field-level rule violations) issues in real-time. Problematic data is published to a dedicated Kafka topic and sunk to S3 for observability. Alerts are sent to stream owners via Slack through the Genchi platform. LLM-based recommendations are provided to simplify semantic rule definition. The solution has been deployed across 100+ Kafka topics.
  5. Data mesh at Grab part I: Building trust through certificationThis post details the implementation of data certification as a cornerstone of Grab's data mesh. It introduces the concepts of data contracts, defining SLAs for schema, data quality, and freshness, and establishes ownership roles (Business Data Owners, Technical Data Owners). A key technical contribution is the automated Data Production Incident (DPI) system, which creates tickets for breached data contracts based on automated data quality tests (availability, timeliness, consistency, completeness, accuracy, validity). The post also highlights the operationalization strategy, including executive buy-in, organization-wide reporting, and dashboards to guide teams on certifying high-usage tables, with a north star metric of 'percentage of queries hitting certified assets'. Outcomes include 75% of queries hitting certified assets, a 400% YoY increase in deprecated assets, and a 58% reduction in P80 datasets.
  7. The complete stream processing journey on FlinkSQLThis post introduces an interactive FlinkSQL solution to enhance stream processing development. It replaces Zeppelin notebooks with a shared FlinkSQL gateway, addressing issues with Flink version maintenance, slow cluster startup times, and integration challenges. The new architecture features a compute layer (FlinkSQL gateway), an integration layer (custom control plane with REST APIs for authentication, authorization, and orchestration), and a query layer (interactive UI with a Hive Metastore catalog). The solution reduces lead time to first results from 5 minutes to 1 minute and provides headless APIs for programmatic integration. Additionally, a portal for FlinkSQL pipeline creation is introduced, enabling low-code development and automated deployment of stream processing pipelines within 10 minutes.
  8. Improving Hugo stability and addressing oncall challenges through automationThis post details the implementation of an automation solution for Grab's Hugo data ingestion platform to improve stability and reduce on-call challenges. The solution introduces six core modules: Signal (collecting failure, SLA, and data quality signals), Diagnosis (determining root cause, assignee, and resolution method), RCA Table (storing diagnosis information), Auto-resolution (executing automated fixes), Data Health API (providing health status to external systems like Kinabalu), and the Hugo pipeline health dashboard (Data Health Workbench) for visualization and manual intervention. The diagnosis module leverages parallelism to handle surges in errors and avoids parsing extensive logs by relying on signals from computation engines. The auto-resolution module employs a flexible framework with custom handlers, including retry mechanisms with backoff. The impact includes enhanced data visibility, timely issue resolution, reduced on-call workload, and improved data contract management.
  9. How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplicationThis post details a custom time-based switchover logic for migrating high-volume real-time streaming traffic between services with zero data loss and duplication. The approach involves moving stream processing code to a shared `/commons` directory, enabling both old and new services to use the same logic. Temporary sink resources are created for validation. A `map[string]time.Time` in the shared code schedules the switchover for each stream, ensuring simultaneous transition. The deployment includes buffer time, close monitoring, and a post-switchover validation period where the old service consumes but does not write. Learnings emphasize the effectiveness of this temporary measure for specific use cases but caution against its general use due to potential rollback inconsistencies.
  10. Unveiling the process: The creation of our powerful campaign builderThis post details the implementation of Counter, Delay, and Limit features within Grab's Trident marketing campaign platform. It explains how these features translate complex campaign logic into executable 'treatments' by leveraging Kafka streams, SQS, and database statistics. The post also describes the development of a node-tree based UI for campaign creation and the algorithm used to convert this tree structure into equivalent treatments, including handling updates to existing campaigns.
  11. PII masking for privacy-grade machine learningIntroduced a CI pipeline to enforce PII tagging in protobuf schemas using keyword scanning and a whitelist. Developed an in-house Flink application to dynamically mask PII in data streams based on schema tags, using HMAC for strings and seeded random generators for numbers/dates. This masked data is then mirrored to a staging environment for ML model tuning.
  12. Using mobile sensor data to encourage safer drivingThis post details how Grab uses telematics data from driver and delivery partners' mobile phones to encourage safer driving. It explains the process of collecting sensor data (accelerometer, GPS) and using custom algorithms to infer vehicle acceleration from phone sensor data, accounting for orientation and data quality variations. The post also describes how unsafe driving instances (braking, acceleration, cornering) are detected using thresholding and how speeding is identified by comparing GPS speeds with map data and aggregated speed statistics. The aim is to proactively detect and mitigate unsafe driving behaviors to reduce accidents and improve user experience.
  13. How we store and process millions of orders dailyDesigned and implemented a dual-database architecture for the Grab Order Platform, separating OLTP (DynamoDB) and OLAP (MySQL RDS) workloads. Optimized DynamoDB usage with primary keys, Global Secondary Indexes (GSIs) with sparse indexing for ongoing orders, and a phased TTL implementation for data retention. Utilized MySQL partitioning for OLAP data retention. Established a Kafka-based data ingestion pipeline with SQS for producer failure handling and back-off retries for consumer consistency, addressing duplicate and out-of-order messages with unique keys and version updates.
  14. Real-time data ingestion in GrabThis post details the implementation of a real-time data ingestion system at Grab, focusing on synchronizing data from MySQL, Aurora, and DynamoDB to Kafka. It covers the architecture, including stream storage (binlogs and DynamoDB streams), event producers (Debezium, Lambda), Kafka, and a Golang stream processor. The post highlights the benefits for data pipelines, business decisions, database replication, and audit trails, emphasizing improved data consistency and reduced database read pressure.
  15. Search indexing optimisationThis post details optimizations to Grab's incremental search data indexing process. It introduces the use of MySQL binary logs (Binlog) for Change Data Capture (CDC) to replace the previous Kafka stream approach. Key optimizations include: 1. Filtering irrelevant stream events by comparing PayloadBefore and PayloadAfter to only process changes relevant to Elasticsearch, reducing DB reads by 90% and filtering 55% of irrelevant events. 2. Performing partial updates in Elasticsearch by using the differences between PayloadBefore and PayloadAfter, reducing DB load by 80% and Elasticsearch load by only sending changed fields. These changes eliminate data loss, reduce coupling with MySQL table structure, and prevent redundant Elasticsearch updates and DB reads.
  16. Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) PredictionsThis post introduces Archivist, a new tracing, statistics, and feedback system for ML/rule-based prediction events. Archivist addresses limitations of previous logging methods (plain text logs, lack of fine-granularity access control, separate Elasticsearch clusters per team) by providing a centralized, performant, and flexible solution. Key improvements include user-defined entities and custom fields, a shared event processor that writes to a central Elasticsearch cluster and S3 data lake, and a user portal for querying, prediction stats, and feedback. It reduces development time for new business scenarios and improves data accessibility for different roles within Grab.
  17. Data First, SLA AlwaysThis post introduces Trailblazer, a Spark Structured Streaming application designed to capture and stream all MySQL binlogs to the data lake. It addresses the challenges of scaling batch ingestion beyond the petabyte threshold by moving to a real-time Change Data Capture (CDC) approach. Key technical details include the use of Debezium on Kafka Connect for capturing binlogs, Spark Structured Streaming for processing, and a custom Redis-based checkpoint management system to ensure stream state restoration and external control over ingestion starting points. The post also discusses scalability considerations for handling varying data volumes and the importance of monitoring data streams using tools like Datadog, Slack, PagerDuty, and an in-house data auditor (DASH). It highlights the ephemeral nature of streams and the need for retry mechanisms, such as those provided by Airflow.
  18. How We Simplified Our Data Ingestion & Transformation ProcessThis post details the evolution of Grab's real-time data ingestion and transformation process. Initially, a system using Kafka for collection, Spark Streaming and a Golang service for transformation, and TalariaDB for storage was built. Challenges arose with operational complexity, Spark Streaming scaling issues, data loss due to S3 eventual consistency, and processing lag. The team simplified the architecture by re-implementing the transformation logic in Golang, addressing event partitioning, ORC encoding, and improving memory utilization. This resulted in a reduced processing lag to approximately 1 minute, fixed data loss issues, and decreased operational workload by removing Spark Streaming.