TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Counter Service Storage Migration

5 posts · Aug 2017 – Jul 2026

What this trail covers

Grab's Counter Service has evolved to provide a self-service platform for fraud detection, enabling data scientists and analysts to create and manage counters without direct engineering intervention. The system is designed for high scalability, handling billions of transactions daily by leveraging Aerospike for time-series data storage with a map-based schema for optimized storage and retrieval. This migration focused on decoupling storage concerns from business logic, redesigning the data model, and optimizing performance by removing a Redis cache and relying on Scylla's native caching. This has led to more balanced load and reduced latency during peak hours.

All Grab trails
  1. Migrating Counter Service storage: Design choices and learningsThis post details the migration of Grab's Counter Service storage from a wide-column database to Aerospike. It covers the redesign of the reader service to abstract storage access using a facade with enum dispatch and configurable operating modes (Single, WithShadow, WithSplit) for gradual rollout and verification. The writer-side data model was redesigned from a row-per-bucket approach to a map-based schema within Aerospike, storing all time-series counts for a counter in a single record. This significantly reduced the record count and primary index memory. The post also discusses the use of `MapIncrementOp` and `MapRemoveByKeyRangeOp` for atomic server-side updates and TTL management, and the challenges encountered with index memory and schema design.
  2. Counter Service: How we rewrote it in RustThe Counter Service, previously using Golang and Scylla/Redis/DynamoDB/MySQL, has been rewritten in Rust. This rewrite achieved 70% infrastructure savings while maintaining similar performance. The team selected Rust due to its efficiency and memory safety, and chose the Counter Service for rewrite due to its 'simple' functionality and high traffic. The rewrite involved using Rust libraries for GRPC (Tonic), web server (Actix-web), Redis (fred.rs), Kafka (kafka-rust), and Scylla (scylla-rust-driver). Internal Go templating was replicated using the Nom parser combinator. Challenges included adapting to the borrow checker and async gotchas, which were managed by initially using clones/Arc and focusing on cooperative scheduling.
  3. Evaluating performance impact of removing Redis-cache from a Scylla-backed serviceThis post details the investigation and removal of a Redis cache from a Scylla-backed counter service. The primary motivation was to eliminate performance spikes and latency issues caused by the cache's 15-minute truncation strategy and 5-minute TTL, which led to cache hit misses and increased Scylla QPS at the end of every 15-minute block. By removing the Redis cache and relying on Scylla's native caching, the service achieved more balanced load distribution and reduced latency, demonstrating a trade-off between external caching complexity and leveraging the underlying database's capabilities.
  4. Using Grab’s Trust Counter Service to Detect Fraud SuccessfullyThis post details the design and implementation of Grab's Counter Service, a self-service platform for fraud detection. It addresses scalability challenges by using ScyllaDB with a multi-buckets strategy for pre-aggregating time-series data into 15min, 1hr, and 1day buckets to improve query performance and meet SLAs. The service offers a UI for counter creation and management, SDKs for backend integration, and a TTL cache layer. It also highlights the role of the Counter Service in ML workflows by providing consistent data sources and counter replay capabilities for historical data simulation.
  5. Migrating Existing DatastoresThis post details the migration of the Grab Identity team's authentication token cache from a single Redis node to an AWS ElastiCache cluster. The team faced challenges with scalability and a single point of failure. They evaluated options including larger Redis instances, custom Redis clusters, and AWS ElastiCache. Ultimately, they opted for a sharded ElastiCache cluster with read replicas to handle high read volumes and provisioned shards to cover future write load growth. A six-step, zero-downtime migration plan was executed using configuration flags, shadow writes, synchronous writes, asynchronous reads with validation, full read switch, and finally, decommissioning the old node. The migration was successful, achieving no downtime, zero collateral damage, and no security loopholes.