Decomposing the GitLab backend database, Part 1: Designing and planning

This post details the initial design and planning phase of decomposing the GitLab backend database. It covers the motivation for moving away from a monolithic Postgres database due to scaling limitations, the exploration of horizontal sharding versus vertical partitioning, and the analysis that identified CI tables as a prime candidate for extraction due to high write traffic. It also outlines the proof-of-concept process, the strategy for managing the 'moving target' of application evolution during the decomposition, and the development of tools like cross-join detection, cross-database transaction detection, query analyzer metrics, and a Rubocop rule to enforce explicit database connections. The post also discusses the use of Rails' multiple database support and the implementation of 'loose foreign keys' to handle inter-database dependencies asynchronously.