TechPrufer
Get started
Replicate
Replicate logoReplicate · Feature trail

Incident Response and Observability

1 post · May 2023

What this trail covers

Replicate has launched a public status page and is publishing detailed incident reports to improve transparency and communicate system health. The first report details a significant outage on May 11th caused by connection pool exhaustion due to stacked INSERT queries on their PostgreSQL database, triggered by new asynchronous prediction update features. The incident highlights the challenges of complex systems failures, the interplay between performance improvements and new hazards, and the need for robust observability and incident response.

All Replicate trails
  1. Status pageThis post details a significant outage on May 11th caused by connection pool exhaustion in their PostgreSQL database. The root cause was identified as stacked INSERT queries, a side effect of newly implemented asynchronous prediction update features. The post explains the background of the issue, the sequence of events leading to the outage, the response taken, and key learnings, emphasizing the complex nature of system failures and the need for careful consideration of performance improvements' impact on reliability. The asynchronous update feature remains disabled while being redesigned.