Status page
5/18/2023
This post introduces a public status page and a detailed incident report for a major outage on May 11th. The report provides a deep dive into the root cause: connection pool exhaustion in PostgreSQL resulting from stacked INSERT queries on prediction IDs. This pattern emerged due to newly implemented asynchronous prediction update features, which reduced latency but inadvertently created contention for unique index locks. The post discusses the system's degraded state, the role of a defense mechanism (asynchronous updates) in both triggering and partially mitigating the failure, and the lessons learned regarding change creating new hazards. It outlines immediate responses, including disabling the asynchronous features and engaging with the database provider. Future work includes redesigning the asynchronous update feature to prevent stacked INSERTs and reviewing database hotspots.