GitLab.com database incident

This post details a critical database incident where production data was lost due to accidental deletion during recovery attempts. It outlines the sequence of events, including spam attacks, database lockups, replication lag, and the accidental wiping of a production database. The post also highlights severe problems encountered with backup and replication mechanisms, including non-functional regular backups, failing pg_dump, disabled Azure disk snapshots for DB servers, and fragile replication procedures. Recovery involved restoring from a staging backup and manually copying data. The incident led to a commitment to improve backup and replication strategies.