TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Highly Concurrent In-Memory Counter

1 post · Oct 2019 – Oct 2025

What this trail covers

Grab's Counter Service has evolved into a self-service platform for fraud detection, enabling data scientists and analysts to create and manage counters without direct engineering intervention. The system is designed for high scalability, handling billions of transactions daily by leveraging Aerospike for time-series data storage with a map-based schema for optimized storage and retrieval. This migration focused on decoupling storage concerns from business logic, redesigning the data model, and implementing a highly concurrent in-memory counter with periodic flushing to reduce database load. The use of Sync.Map in GoLang has shown significant performance improvements over traditional mutex-based approaches for high-concurrency, read-heavy workloads.

All Grab trails
  1. Highly concurrent in-memory counter in GoLangThis post details the implementation and benchmarking of a highly concurrent in-memory counter in GoLang for tracking marketing campaign usage. It compares a mutex-based approach with the use of `sync.Map`, demonstrating that `sync.Map` offers a 3x speed advantage and significantly higher throughput for read-heavy workloads with a finite set of frequently accessed keys. The flushing mechanism was also optimized by atomically removing keys from the map using `LoadAndDelete` instead of locking the entire map. This optimization resulted in a 68% decrease in usage tracking update queries and substantial reductions in database CPU utilization.
  2. Counter Service: How we rewrote it in RustThis post details the rewrite of Grab's Counter Service from Golang to Rust. The rewrite was motivated by the potential for infrastructure cost savings and improved performance. The team selected the service based on its 'simple' functionality and high traffic volume. The approach involved a black-box rewrite, focusing on API contracts and data consistency. Key challenges included adapting to Rust's borrow checker and async programming model, as well as finding suitable OSS libraries for dependencies like Datadog and Redis. The rewrite resulted in a 70% infrastructure saving while maintaining similar performance.
  3. Using Grab’s Trust Counter Service to Detect Fraud SuccessfullyThis post details the initial design and challenges of Grab's Counter Service, focusing on its role in fraud detection. Key contributions include: 1. Establishing the need for a self-service platform to manage counters for fraud detection, overcoming the limitations of conventional methods. 2. Designing a system with asynchronous data ingestion and synchronized transactions, utilizing Grab-Stats (ScyllaDB) for scalable time-series data storage with a p99 read latency under 150ms. 3. Implementing a multi-bucket strategy to improve query performance by pre-aggregating signals into different time granularities (15min, 1 hour, 1 day) to satisfy SLAs for both short-term and long-term time windows. 4. Providing a UI-based rule editor and SDKs for engineers to facilitate self-service counter creation and integration. 5. Highlighting the role of counters as features for ML models and the service's ability to provide historical data sets via a counter replay feature.