TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Redis Data Consistency and Replication

3 posts · Jan 2018 – Sep 2020

What this trail covers

Grab has evolved its Redis data consistency and replication strategies by addressing issues where pipeline calls could crash Redis clusters. This involved analyzing the Go-Redis client's behavior during node unreachability, specifically how it maps commands to nodes and the delay in state updates. The team identified that a single command failure within a pipeline could cause the entire pipeline to fail, and that the client's one-minute state refresh interval led to prolonged unavailability when nodes were patched for security vulnerabilities like Meltdown. To mitigate performance impacts, Grab added additional shards to clustered Redis instances and provisioned more nodes or switched to Redis Cluster for non-clustered instances. For systems requiring significant code changes, service teams implemented data sharding across multiple nodes. These mitigations were performed over 24 hours to ensure no consumer-facing impact during peak traffic.

All Grab trails
  1. Uncovering the Truth Behind Lua and Redis Data ConsistencyThis post details the investigation and resolution of Redis master/replica data inconsistency caused by Lua scripts executing differently on master and replica nodes, leading to divergent hash encodings (hashtable vs. ziplist). The root cause was identified as Redis's behavior of executing Lua scripts independently on replicas and the potential for different hash encodings during RDB loading. The solution implemented was to ensure deterministic Lua script behavior by sorting the outputs of HKEYS/HGETALL, preventing data divergence and high CPU usage on replicas.
  2. Preventing Pipeline Calls from Crashing Redis ClustersThis post details an investigation into a Redis cluster outage caused by a single slave node failure. It explains how the Go-Redis client's pipeline error handling and one-minute state refresh interval contributed to a prolonged service disruption. The post identifies that a single command failure within a pipeline could cause the entire pipeline to fail and that the client's state refresh delay led to continued attempts to contact unreachable nodes. It also clarifies that write queries were unaffected due to master node direct routing and client-side monitoring.
  3. Dealing with the Meltdown Patch at GrabThis post details the performance impact of the Meltdown patch on Grab's Elasticache Redis nodes, characterized by sudden, significant spikes in CPU utilization. The team initially investigated workload changes but escalated to AWS when multiple unrelated nodes exhibited the same behavior, including replica nodes. Failovers to unpatched replicas provided temporary relief. Upon understanding the issue was due to rolling patches by AWS, Grab implemented horizontal scaling strategies: adding shards to clustered Redis and provisioning additional nodes or switching to Redis Cluster for non-clustered Redis. For some non-clustered Redis instances, service teams made code changes to shard data across multiple nodes. These actions were completed within 24 hours to ensure minimal consumer impact during peak traffic.