Decomposing the GitLab backend database, Part 3: Challenges and surprises

This post details specific challenges and surprises encountered during the production migration of GitLab's monolithic Postgres database to a decomposed architecture. It highlights issues with autovacuum on large CI tables delaying write blocking triggers, the need to track down and close long-running SSH database connections, the oversight of certain cron jobs, and an overload on the Sentry server due to client-side error tracking during downtime. It also discusses the impact of cascading replication on CI replica latency and the intricate process of incrementally re-balancing PGBouncer connections without saturating the main Postgres database, which required small, carefully analyzed steps during low utilization periods. The post concludes by noting that despite the added complexity of an additional database, developer productivity has remained largely unaffected due to Rails model abstraction, and the project has provided significant scalability headroom.