Post-Incident Review on the Atlassian April 2022 outage

This post details a major incident where a misconfiguration led to the deletion of 775 customer sites, impacting 775 customers for up to 14 days. The root cause was a communication gap and insufficient system warnings during the execution of a script to delete a legacy Atlassian app. The script incorrectly received site IDs instead of app IDs, leading to site deletion. The response involved a cross-functional incident management team working 24/7 to restore sites. Preventative measures include implementing universal 'soft deletes', accelerating the Disaster Recovery (DR) program for large-scale deletion events, revising incident management processes for large-scale incidents, and creating a large-scale incident communications playbook.