Status page
May 18, 2023

Summary
This post details a significant outage on May 11th caused by connection pool exhaustion in their PostgreSQL database. The root cause was identified as stacked INSERT queries, a side effect of newly implemented asynchronous prediction update features. The post explains the background of the issue, the sequence of events leading to the outage, the response taken, and key learnings, emphasizing the complex nature of system failures and the need for careful consideration of performance improvements' impact on reliability. The asynchronous update feature remains disabled while being redesigned.