Postmortem of database outage of January 31

This post details a major database outage caused by accidental data removal during an attempt to restore replication after WAL segment loss. It highlights critical failures in backup procedures (pg_dump version mismatch, DMARC rejection of error emails), reliance on LVM snapshots for DR, and the lack of documented procedures for pg_basebackup. The post also describes the immediate recovery steps and outlines future improvements to operations and recovery procedures, including implementing WAL archiving, improving backup verification, and enhancing runbooks.