Database Architecture and Specialization

4 posts · Sep 2016 – Aug 2022

What this trail covers

GitLab is decomposing its monolithic PostgreSQL database into separate 'main' and 'ci' clusters to improve scalability and reliability. This involves splitting CI-related data into its own cluster, allowing for independent tuning and increasing database capacity. The process includes phased rollouts, extensive testing, and the development of tools to manage multi-database environments. This decomposition is planned for self-managed instances in the future. This post discusses the architectural c

All GitLab trails

Aug 2022GitLab database decomposition: Final migration and resultsThis post details the final migration phase of GitLab's database decomposition, focusing on the decision between zero-downtime and full-downtime approaches. It outlines the chosen two-hour downtime migration strategy, emphasizing the importance of extensive rehearsals (seven on staging) and a straightforward rollback procedure. The post quantifies the results of the migration, showing reduced CPU utilization on the primary database, significant storage savings, decreased dead tuples and vacuuming saturation, and a substantial reduction in average query duration for Sidekiq workloads.
Jun 2022We are splitting our database into Main and CIThis post details the planned split of GitLab.com's single PostgreSQL database into 'main' and 'ci' clusters. It explains the motivation (scaling writes from CI features), the technical approach (functional decomposition), the expected impact (increased performance, stability, and shipping velocity), and the preparation steps taken (phased rollout, CI write traffic split, multi-database tooling). It also outlines the production change process and rollback strategies.
Apr 2022Two sizes fit most: PostgreSQL and ClickhouseThis post discusses the architectural considerations of using specialized databases like PostgreSQL for OLTP and ClickHouse for OLAP, highlighting the limitations of a 'one-size-fits-all' approach and the benefits of choosing the right tool for the job. It references historical database research and current industry trends, contrasting traditional row-oriented OLTP databases with column-oriented OLAP databases.
Jun 2019Why we're ending support for MySQL in 12.1GitLab is ending support for MySQL in version 12.1. This decision was made due to MySQL's limitations for GitLab's specific use cases, including performance issues with nested groups, limitations on column sizes, lack of support for TEXT type columns without length specification, and absence of partial indexes. These limitations also impacted features like Geo. Supporting MySQL required complex, backend-specific code, increasing merge request complexity and slowing down development velocity. CI systems also ran test suites twice, increasing costs. By removing MySQL support, GitLab can leverage backend-specific capabilities like PostgreSQL's LATERAL JOIN. Usage Ping data indicated a strong customer preference for PostgreSQL, with a significant majority of instances already migrated. The post provides migration documentation for users still on MySQL.
Dec 2018Introducing Patroni as the Postgres Failover Manager on GitLab.comGitLab.com is deploying Patroni as its Postgres Failover Manager. This involves a two-weekend maintenance window for a dry run and the actual cutover. During the cutover, GitLab.com services including the website, Git SSH/HTTPS, registry, CI/CD, and Pages will be unavailable for approximately 30 minutes.
Advertisement
Oct 2017Scaling the GitLab databaseThis post details GitLab's initial steps in scaling its single PostgreSQL database setup. It explores four techniques: application code optimization, connection pooling, load balancing, and sharding. The post focuses on connection pooling using pgbouncer and database load balancing using hot-standby replicas. It argues against sharding due to low write load and high implementation complexity. The implementation of pgbouncer reduced active PostgreSQL connections significantly and allowed for a lower `max_connections` setting.
Sep 2016GitLab Infrastructure UpdateThis post details the challenges GitLab.com faced due to scale, specifically around storage (CephFS cluster implementation) and PostgreSQL performance issues (vacuuming impact). It highlights the lack of detailed documentation for scaling problems and emphasizes GitLab's commitment to sharing their solutions.