We took down production by misconfiguring our ETL

This post details a production incident caused by an ETL misconfiguration leading to a PostgreSQL WAL filling up the disk. It explains the technical details of WAL growth, the limitations of RDS auto-scaling during optimization, and the complexities of using a read replica for failover. It proposes specific technical improvements such as better alerting, WAL size limits, and moving ETL off the production DB.