Dealing with the Meltdown Patch at Grab

This post details the performance impact of the Meltdown patch on Grab's Elasticache Redis nodes, characterized by sudden, significant spikes in CPU utilization. The team initially investigated workload changes but escalated to AWS when multiple unrelated nodes exhibited the same behavior, including replica nodes. Failovers to unpatched replicas provided temporary relief. Upon understanding the issue was due to rolling patches by AWS, Grab implemented horizontal scaling strategies: adding shards to clustered Redis and provisioning additional nodes or switching to Redis Cluster for non-clustered Redis. For some non-clustered Redis instances, service teams made code changes to shard data across multiple nodes. These actions were completed within 24 hours to ensure minimal consumer impact during peak traffic.