TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Spark Observability Platform with StarRocks

6 posts · Mar 2019 – Mar 2025

What this trail covers

Grab has evolved its Spark observability by migrating from the TIG stack (Telegraf, InfluxDB, Grafana) to a unified platform leveraging StarRocks. This new architecture addresses fragmented user experience, operational overhead, and data management challenges by using StarRocks as a single source for real-time and historical data, enabling direct Kafka ingestion, and providing a custom web application (Iris UI) for a unified, user-friendly interface. Superset integration is maintained for real-time analysis. This post details the adoption of structured logging to improve the observability of Grab's backend systems, focusing on the implementation of a new Golang library that supports dynamic log levels, consistent JSON structuring via the Common Grab Log Schema (CGLS), and improved log correlation and causal ordering. This initiative aims to reduce Mean Time To Resolve (MTTR) by making logs more queryable and understandable, and to manage logging costs more effectively by enabling dynamic log level adjustments and focusing structure on essential fields.

All Grab trails
  1. Building a Spark observability product with StarRocks: Real-time and historical performance analysisThis post details the evolution of Grab's Spark observability platform from the Iris tool (TIG stack) to a new architecture centered around StarRocks. It outlines the pain points of the previous system (fragmented UX, operational overhead, data management issues with InfluxDB) and how StarRocks addresses them with full SQL compatibility, direct Kafka ingestion, flexible materialized views, efficient handling of high-cardinality data and joins, and native S3 integration. The new system architecture, data model, and ingestion process using StarRocks' routine load feature are described, including table schemas for cluster metadata, worker metrics, and Spark metrics.
  2. Ensuring data reliability and observability in risk systemsThis post details the implementation of Apache Flink SQL for real-time data aggregation and transformation of heterogeneous JSON data from various upstream services feeding into the GrabDefence risk management platform. It describes the use of custom table functions like JSONEXPLOAD to deconstruct nested JSON, 5-minute tumbling windows for aggregation, and the integration of Flink's output with Datadog for anomaly detection and alerting via Slack. The post also covers strategies for organizing Datadog monitors by service stream and leveraging its Anomaly Detection feature to quickly identify data quality issues.
  3. Iris - Turning observations into actionable insights for enhanced decision makingThis post introduces Iris, a new observability platform for ETL jobs, specifically focusing on Spark and Presto. Iris captures critical job metrics at the JVM level, including runtime, CPU/memory utilization, garbage collection, and stage/task execution details. It supports both real-time monitoring via a TIG stack (Telegraf, InfluxDB, Grafana) and offline analytics by pushing metrics to the data lake. The platform's architecture includes a Metrics Collector module that uses Uber's JVM Profiler and sparkMeasure to gather detailed metrics without requiring application code changes. This provides granular insights into resource consumption, enabling better provisioning and cost optimization.
  4. How We Built a Logging Stack at GrabThis post details the initial build of Grab's logging stack using Elasticsearch, Logstash, and Kibana (ELK Stack). It covers the problem statement of lacking visibility into service logs, the challenges of handling large data volumes (25TB/day), the design decisions for a multi-petabyte cluster including node roles (ingest, coordinator, master, data) and the choice of ingest nodes over Logstash, and the importance of monitoring (Datadog integration). It also highlights pitfalls encountered, such as Field Data Cache exhaustion due to sorting by '_id', the impact of Translog Compression on shard transfers, issues with Segment Memory allocation due to too many shards per node, and the importance of Index Mapping and picking the right Analyser.
  5. Structured Logging: The Best Friend You’ll Want When Things Go WrongThis post details the development and adoption of a new structured logging library in Golang to improve observability and reduce MTTR. Key contributions include: 1. Dynamic Log Levels: Enabling runtime adjustment of log levels via configuration management to control verbosity and cost. 2. Consistent Structure in Logs: Implementing the Common Grab Log Schema (CGLS) to enforce deterministic JSON formatting with essential fields for efficient indexing in Elasticsearch, while other fields are placed in a single unstructured field. 3. Plug and Play Support with Grab-Kit: Easy integration with the existing microservice framework. 4. Improved Log Correlation: Automatic inclusion of request IDs and trace IDs to link logs across services. 5. Causal Ordering: Preserving log generation time instead of write time for accurate event sequencing. The library aims to address issues with previous syslog-style logging, high costs, and limitations in leveraging the Elastic stack.