GitLab.com database incident

What this post added

This post details a severe database incident on GitLab.com where six hours of data (issues, merge requests, users, comments, snippets) were lost due to a combination of spam attacks, replication lag, and a critical human error during recovery attempts. It highlights the failure of multiple backup and replication techniques (LVM snapshots, regular backups, pg_dump, Azure disk snapshots, S3 backups) and the fragility of the replication procedure. The recovery involved restoring from a staging database backup. The incident underscores the urgent need for robust, reliable, and well-documented backup and replication strategies.