All Hands on Deck

This post details the incident response process during a major outage on May 12th, 2020, highlighting the 'All Hands on Deck' approach, the transition from a heroic on-call model to a FEMA-inspired Incident Command System, and the specific steps taken to diagnose and mitigate a HAProxy configuration issue that led to a full service disruption. It emphasizes the importance of structured incident response, learning from failures, and implementing action items to prevent recurrence.