High Availability

5/7/2026

This post announces the general availability of Gitaly on Kubernetes, a significant operational improvement for teams running GitLab on Kubernetes. It details the technical challenges encountered in running Gitaly within Kubernetes, specifically around cgroup management and pod restarts. Solutions implemented include using an init container to make cgroupfs writable and configuring client retries to mitigate downtime during pod replacements. Benchmark results demonstrating high success rates for Git operations are provided, comparing Gitaly on VMs to Gitaly on Kubernetes. The post also mentions ongoing work to bring Gitaly Cluster (Praefect) support to Kubernetes.

2/18/2026

Introduced a 99.9% availability SLA with service credits for Ultimate customers on GitLab.com and GitLab Dedicated. Defined covered experiences (Issues, merge requests, Git operations, Container Registry, Package Registry, API requests) and excluded experiences. Detailed the definition of 'downtime minutes' as periods with degraded availability of 5% or more of valid customer requests leading to server errors (HTTP 5xx or connection timeouts > 30s). Outlined the process for customers to claim service credits.

11/30/2021

This post introduces GitLab Incident Management as a built-in feature of the DevOps Platform. It details the costs of downtime, defines incidents and incident management, and explains the benefits of managing incidents within GitLab due to the co-location of code, CI/CD, and monitoring tools. It outlines the process of creating incidents manually or automatically via webhooks (e.g., PagerDuty), managing alerts, setting up on-call schedules, defining escalation policies, and the current notification methods (email, with plans for Slack, SMS, and phone calls).

4/29/2021

Introduced a new Geo replication and verification framework to reduce code duplication and enable easier addition of new data types. This involved designing a `Replicator` class and `Replicator strategies` for blobs and Git repositories. Implemented initial support for replicating new Package files as a proof of concept, laying the foundation for future data type replication and verification.

3/17/2021

This post details the challenges and learnings from testing the 'Maintenance Mode' feature, which impacts all parts of the system. It highlights the success of a crowd-sourced testing approach involving multiple development stages, emphasizing iterative and collaborative testing, cross-team contributions, clear communication, and the value of documentation for developers.

1/21/2021

This post introduces Praefect, a new component designed to manage traffic to Gitaly storage nodes and enable Gitaly Clusters for high availability. It details the evolution from NFS to Gitaly, the challenges of scaling Git for high availability, and the design choices leading to Praefect. The post covers the initial implementation of eventual consistency with three data copies and the subsequent development of strong consistency using a three-phase commit protocol integrated with Git hooks. It also discusses cost considerations, the introduction of read distribution, and the challenges and lessons learned during the development and rollout of Praefect, including the decision to make it a required component.

1/14/2021

This post details a specific HAProxy configuration issue encountered during a rollout, where the `server-state-file` directive caused the port configuration to be overwritten, leading to traffic being sent to the wrong port and causing TLS handshake failures. A workaround was implemented by changing the backend server name to bypass the state file's port persistence.

4/16/2020

This post details the enablement of GitLab Geo on the GitLab.com staging environment. It covers the rationale for testing Geo at scale, the setup process including selective synchronization and Gitaly shard configuration, and challenges encountered with PostgreSQL 9.6 and Foreign Data Wrappers. It also highlights bug fixes and future plans for end-to-end test automation and PostgreSQL HA configurations on Geo nodes.

2/13/2019

This post details the implementation and benefits of using delayed replication with PostgreSQL for disaster recovery. It explains how a delayed replica, configured with `recovery_min_apply_delay` and using archive recovery with `wal-e`, can be used for point-in-time recovery from accidental data deletions. The process involves pausing the replica, adjusting recovery targets to a specific time, and then exporting/importing recovered data. It highlights the advantages over cold backups by eliminating the need to restore a full basebackup and reducing the WAL replay window.

9/14/2018

This post details the initial architectural decisions and implementation phases for GitLab Geo. It covers the MVP phase, which focused on database replication, API calls for disk changes, and using Git for repository replication. It also outlines the decision to authenticate on the primary node only and use OAuth for SSO/SSO. The first-generation synchronization mechanism leveraged GitLab's webhook system for improved notification and security.

12/8/2016

Introduced GitLab Enterprise Edition Premium, which bundles previously separate enterprise products into a single subscription. Key features include Premium Support (4-hour response, 24/7 emergency support), GitLab Geo for read-only mirrors to improve remote team productivity with large repos, and File Locking for managing binary/text files. Also introduced a Pivotal Tile for easier deployment on Pivotal Cloud Foundry. Renamed the existing enterprise plan to GitLab Enterprise Edition Starter.

5/10/2016

Introduced the 'Push to remote repository' functionality for GitLab Enterprise Edition, enabling users to push changes from a GitLab repository to an external repository. This complements the existing 'sync from remote repository' feature, allowing GitLab to act as a central hub for development with synchronization to other services. The setup is configurable in Settings -> Mirror Repository and can be triggered manually or automatically every hour. An experimental chain of three repositories (Bitbucket -> GitLab -> GitHub) was demonstrated to showcase the functionality.

9/1/2015

This post details a specific incident of GitLab.com being offline for an hour due to perceived filesystem corruption on an NFS server. It describes the initial misdiagnosis based on `dmesg` output, the decision to take the site offline for `fsck`, and the subsequent realization that the errors were recurring daily and not the cause of the immediate slowdown. The post highlights the challenges of diagnosing NFS issues, the decision to abort `fsck` and bring the site back online, and the ongoing effort to migrate data off the problematic filesystem. It also reflects on lessons learned regarding communication during crises and the need for better status updates.

6/4/2015

This post details a specific outage on GitLab.com due to an infrastructure upgrade that moved to a clustered setup. The root cause was identified as a backup script hanging during the filesystem freeze/unfreeze process on the backend server, which prevented workers from mounting the NFS share. The extended downtime was attributed to insufficient training and documentation for on-call engineers regarding the new infrastructure. Next steps included removing the filesystem freeze from the backup script, implementing a secondary backup strategy for SQL data, and initiating regular operations drills for on-call engineers.

1/28/2015

This post details the process of applying OS-level security patches (glibc vulnerability) to production systems with minimal downtime by leveraging the existing DRBD-based clustered setup for fast failovers. It outlines the steps of updating OS packages and rebooting servers, and specifically highlights how the DRBD-based clustered setup enabled a failover in under a minute to apply updates to the standby server, minimizing downtime for gitlab.com.

9/22/2014

This post announces a partnership with WANdisco to integrate Git MultiSite with GitLab Enterprise Edition. This integration leverages WANdisco's active-active replication and Distributed Coordination Engine (DConE) to provide 100% uptime and LAN-speed performance for distributed development teams by turning servers into writable Git peer nodes and resolving conflicts transparently.

12/17/2013

This post introduces an overview of GitLab High Availability Solutions, aiming to address customer concerns about continuous availability for business-critical services. It highlights the trade-off between uptime, complexity, and cost, and outlines a spectrum of solutions from basic server backups to clustered/master-master filestore configurations. The post serves as an initial step in considering HA options rather than a direct setup guide.