March 1st Partial Database Outage

This post details a critical incident where a bug in an inactivity-checking script, combined with an inappropriate deployment process, led to the accidental deletion of 515 free-tier indexes. The root cause was a faulty SQL query and the use of a lighter deployment process for a data-modifying script. The incident response involved scaling down operators, engaging cloud provider support for recovery, and a multi-day restoration process. Lessons learned emphasize the need for a rigorous database culture, specific design considerations for the free plan, improved testing and shipping processes for critical features, and the prioritization of soft deletions and data exports.