TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Cache Sharding and Scaling

2 posts · Mar 2017 – Jun 2017

What this trail covers

Grab's Common Data Service (CDS) has evolved its caching strategy to horizontally scale its Redis-based cache system. This involved implementing a custom sharding mechanism to distribute cache keys across multiple ElasticCache instances, addressing the limitations of single-threaded Redis and the potential bottleneck of a centralized proxy like Twemproxy. The new design allows for increased caching capacity and improved P99 latency under heavy load. This post details an investigation into Redis key expiration consistency issues on read replicas, revealing that slaves do not expire keys unless explicitly told by the master, and that passive expiration is too slow at scale. The findings led to recommendations to upgrade Redis versions more frequently and pay closer attention to cache invalidation strategies during software design.

All Grab trails
  1. How We Scaled Our Cache and Got a Good Night's SleepThis post details the technical decision-making process for scaling Grab's cache system. It outlines the problem of a growing load on the single-threaded Redis cache, evaluates three potential solutions (master-slave, Twemproxy, custom sharding), and explains the chosen custom sharding approach. The implementation involved creating a `ShardedCache` interface in Go, compositing it with the existing `Cache` interface, and utilizing consistent hashing. The deployment strategy involved a two-day phased rollout with double-writing to the new system and cache population via a cron job to ensure reliability and minimize disruption. The outcome demonstrated improved P99 latency and better resource utilization.
  2. A Key Expired in Redis, You Won't Believe What Happened NextThis post details an investigation into Redis key expiration consistency issues on read replicas in a Redis 2.x ElastiCache setup. It reveals that slaves do not expire keys unless explicitly told by the master, and that the passive expiration mechanism is too slow at scale (taking over 110 hours to guarantee no expired keys for a dataset of ~5.6 million keys). The findings led to recommendations to upgrade Redis versions more frequently and pay closer attention to cache invalidation strategies during software design.