Postmortem of database outage of January 31

What this post added

This post details a major database outage on GitLab.com caused by accidental data removal from the primary database server. It outlines the incident's timeline, the root cause (an engineer errantly wiping the primary database directory while attempting to restore replication), and the subsequent data loss. The post also critically analyzes the broken recovery procedures, specifically highlighting failures in `pg_dump` backups due to version mismatches and lack of DMARC for cronjob error notifications, and the unsuitability of Azure disk snapshots and LVM snapshots for disaster recovery of the database. It concludes with a commitment to improving operations and recovery procedures to prevent future occurrences.