TechPrufer
Sign inGet started

Blogs›GitLab›Database Architecture and Specialization

Database Architecture and Specialization

Database Architecture and Specialization

4
posts
2017–2022

GitLab has evolved its database architecture by decomposing the monolithic Postgres database into independent databases, starting with the separation of CI-related data. This involved extensive planning, proof-of-concept work, and the development of tools and documentation to manage cross-database dependencies, such as cross-join detection, cross-database transaction detection, and loose foreign keys. The project leveraged Rails' multi-database support and addressed challenges like mirroring dat

2022

Two sizes fit most: PostgreSQL and Clickhouse

4/29/2022

This post analyzes the historical and current landscape of database technologies, contrasting the general-purpose OLTP capabilities of PostgreSQL with the specialized OLAP capabilities of ClickHouse. It delves into the architectural differences between row-oriented and column-oriented storage, the evolution of in-memory databases, and the reasons why a single database architecture is not suitable for all workloads. The post also references GitLab's own migration from MySQL to PostgreSQL, underscoring the practical considerations for choosing database systems.

2019

Why we're ending support for MySQL in 12.1

6/27/2019

GitLab is ending support for MySQL in version 12.1. This decision was made due to MySQL's limitations for GitLab's specific use cases, including performance issues with nested groups, limitations on column sizes, lack of support for TEXT type columns without length specification, and absence of partial indexes. These limitations also impacted features like Geo. Supporting MySQL required complex, backend-specific code, increasing merge request complexity and slowing down development velocity. CI systems also ran test suites twice, increasing costs. By removing MySQL support, GitLab can leverage backend-specific capabilities like PostgreSQL's LATERAL JOIN. Usage Ping data indicated a strong customer preference for PostgreSQL, with a significant majority of instances already migrated. The post provides migration documentation for users still on MySQL.

2018

Introducing Patroni as the Postgres Failover Manager on GitLab.com

12/5/2018

GitLab.com is deploying Patroni as its Postgres Failover Manager. This involves a two-weekend maintenance window for a dry run and the actual cutover. During the cutover, GitLab.com services including the website, Git SSH/HTTPS, registry, CI/CD, and Pages will be unavailable for approximately 30 minutes.

2017

Scaling the GitLab database

10/2/2017

This post details GitLab's initial steps in scaling its single PostgreSQL database setup. It explores four techniques: application code optimization, connection pooling, load balancing, and sharding. The post focuses on connection pooling using pgbouncer and database load balancing using hot-standby replicas. It argues against sharding due to low write load and high implementation complexity. The implementation of pgbouncer reduced active PostgreSQL connections significantly and allowed for a lower `max_connections` setting.