ETL Incident Response

This post details a significant production outage caused by a misconfiguration in the ETL process, specifically related to the PostgreSQL Write-Ahead Log (WAL) growing unboundedly. It covers the root cause (Fivetran connection failure leading to WAL buildup), the impact of auto-scaling limitations during disk optimization, and the challenges encountered when attempting to use a read replica as a failover. The post outlines immediate next steps including better alerting on disk space and Fivetran failures, improving WAL handling (e.g., WAL size limits), and moving non-essential workloads off the production database. It emphasizes the importance of robust incident response and learning from failures.