Slack’s Outage on January 4th 2021

This post details the incident response to Slack's outage on January 4th, 2021. It describes the root cause as network saturation in AWS Transit Gateways, exacerbated by a post-holiday traffic surge. The incident revealed dependencies of monitoring services on the same network infrastructure, leading to a loss of visibility during the outage. It also details issues with autoscaling and instance provisioning under degraded network conditions, specifically the overloading of the 'provision-service' due to increased latency and resource usage. The post outlines the steps taken to recover, including disabling downscaling, manual upscaling attempts, and the eventual recovery as AWS increased TGW capacity. Post-incident actions include architectural changes to isolate monitoring services, preemptive upscaling of TGWs, and reevaluation of health-checking and autoscaling configurations, along with load testing of critical services.