Slack’s Incident on 2-22-22

This post details a major incident on 2-22-22 that exposed cascading failure scenarios within Slack's infrastructure. It highlights the complex interactions between the client boot process, Vitess datastores, caching (Memcached/Mcrouter/Mcrib), and service discovery (Consul). The incident was triggered by Consul agent restarts impacting Mcrib's ability to maintain cache stability, leading to cache misses and subsequent database overload due to inefficient scatter queries for GDM membership. Mitigations involved aggressive throttling of client boot requests and a more granular approach to increasing limits. The post explains the tipping point where cache misses lead to database overload and prevent cache refilling, emphasizing the need for robust cache management and efficient data querying strategies.