TechPrufer
Get started
GitLab
GitLab logoGitLab · Feature trail

High Availability

17 posts · Dec 2013 – May 2026

What this trail covers

GitLab Geo is evolving its replication framework to enable teams across GitLab to add new data types with reduced duplication and effort. This includes a new Replicator class and Replicator strategy concept for blobs and Git repositories, and a more flexible event system. The goal is to minimize lag time for Geo support, increase awareness in other teams, and influence the planning of new features to be Geo-friendly from the start. Initial implementation focused on replicating new Package files. GitLab now backs its 99.9% availability commitment with service credits for Ultimate customers on GitLab.com and GitLab Dedicated. When monthly availability falls below this threshold, eligible customers receive credits toward future invoices. This commitment ensures your DevSecOps workflows have the reliability they need.

All GitLab trails
  1. Consolidate your GitLab stack with Gitaly on KubernetesThis post announces the general availability of Gitaly on Kubernetes, a significant operational improvement for teams running GitLab on Kubernetes. It details the technical challenges encountered in running Gitaly within Kubernetes, specifically around cgroup management and pod restarts. Solutions implemented include using an init container to make cgroupfs writable and configuring client retries to mitigate downtime during pod replacements. Benchmark results demonstrating high success rates for Git operations are provided, comparing Gitaly on VMs to Gitaly on Kubernetes. The post also mentions ongoing work to bring Gitaly Cluster (Praefect) support to Kubernetes.
  2. GitLab backs 99.9% availability SLA with service creditsIntroduces a 99.9% availability SLA with service credits for Ultimate customers on GitLab.com and GitLab Dedicated. Defines 'downtime minutes' as degraded availability of 5% or more of valid customer requests for covered experiences (Issues, merge requests, Git operations, Container Registry, Package Registry, API requests) resulting in server errors (HTTP 5xx or connection timeouts > 30 seconds). Outlines the process for claiming service credits.
  3. Downtime happens, but GitLab Incident Management can helpThis post introduces GitLab Incident Management as a built-in feature of the DevOps Platform. It details the costs of downtime, defines incidents and incident management, and explains the benefits of managing incidents within GitLab due to the co-location of code, CI/CD, and monitoring tools. It outlines the process of creating incidents manually or automatically via webhooks (e.g., PagerDuty), managing alerts, setting up on-call schedules, defining escalation policies, and the current notification methods (email, with plans for Slack, SMS, and phone calls).
  4. How we are closing the gap on replicating *everything* in GitLab GeoIntroduced a new Geo replication and verification framework to reduce code duplication and make it easier for non-Geo engineers to add support for new data types. This framework includes a `Replicator` class and `Replicator strategy` concept for categorizing data like blobs and Git repositories. The event system was made more flexible and lightweight. The initial implementation focused on replicating newly created Package files, excluding replication of changes, deletes, backfills, and verification for the first iteration.
  5. How we tested a feature that affected (almost) all parts of GitLabThis post details the challenges and learnings from testing the 'Maintenance Mode' feature, which impacts nearly all parts of GitLab. It highlights the need for crowd-sourced testing by involving engineers from all 13 stages to identify critical features for prioritization. Key learnings include the importance of iterative and collaborative testing, asking other teams for contributions, clear communication, recognizing documentation's role in developer communication, iterating on discussions, and involving more people in the process.
  7. Meet Praefect: The traffic manager making your Git data highly availableThis post introduces Praefect, a new component designed to provide high availability for Git data by acting as a proxy to Gitaly storage nodes. It details the evolution from eventual consistency (three copies, read-only access to secondaries) to strong consistency (all three copies always up to date, three-phase commit transactions via Git hooks). It also discusses the cost implications of replication and the upcoming features like read distribution and variable replication factor. Challenges and lessons learned regarding optional components and early production deployment are also highlighted.
  8. This SRE's HAProxy Config Change: An Unexpected JourneyThis post details a debugging journey involving an HAProxy configuration change for the gitlab-pages PROXYv2 project. The core issue was the `server-state-file` directive in HAProxy, which persisted the old port number (1443) even after the configuration was updated to use a new port (2443) for PROXYv2 traffic. This led to HAProxy attempting to send PROXYv2 traffic to the TLS port, causing connection failures. The workaround involved changing the backend server name to bypass the state file's port override. The post also explains the discovery of the PROXY protocol v2 header signature.
  9. Why we enabled Geo on the staging environment for GitLab.comThis post details the enablement of GitLab Geo on the staging environment for GitLab.com. It describes the motivation for testing Geo at scale, the setup process including selective synchronization of the 'gitlab-org' group, configuration of Gitaly shards, and the use of cascading streaming replication with Patroni. Challenges encountered include PostgreSQL 9.6 limitations with FDW queries and the need for increased statement timeouts. Bug fixes for synchronization issues were also identified and addressed. Future plans include enabling end-to-end tests on staging and testing high-availability configurations for PostgreSQL on Geo nodes.
  10. How we used delayed replication for disaster recovery with PostgreSQLThis post details the implementation and benefits of using delayed replication with PostgreSQL for disaster recovery. It explains how a delayed replica, configured with `recovery_min_apply_delay` and using archive recovery with `wal-e`, can be used for point-in-time recovery from accidental data deletions. The process involves pausing the replica, adjusting recovery targets to a specific time, and then exporting/importing recovered data. It highlights the advantages over cold backups by eliminating the need to restore a full basebackup and reducing the WAL replay window.
  11. What's up with GitLab.com? Check out the latest data on its stabilityThis post details the migration of GitLab.com from Azure to Google Cloud Platform (GCP), highlighting the resulting improvements in availability and performance. It quantifies the availability increase (99.68% to 99.88%) and reduction in downtime (32 mins/week to 13 mins/week). Performance improvements are illustrated with latency histogram charts for overall, API, and Git requests. The post attributes these gains to a more powerful Gitaly fleet on GCP with significantly better IO performance (SSD vs. magnetic disks, higher IOPS and bandwidth) and a reduction in 'unicorn worker saturation' errors through strategies like aggressive timeouts and circuit breakers.
  12. How we built GitLab GeoThis post details the initial architectural decisions and implementation phases for GitLab Geo. It covers the MVP phase with decisions on using native database replication, API calls for disk changes, Git for repository replication, and a primary-node-only authentication model using OAuth. It also discusses not replicating Redis data or user attachments initially. The first-generation synchronization mechanism leveraged webhooks for change notifications, improving upon the MVP's custom API.
  13. Feature highlight: Push to a remote repositoryIntroduced the 'Push to remote repository' functionality for GitLab Enterprise Edition, enabling users to push changes from a GitLab repository to an external repository. This complements the existing 'sync from remote repository' feature, allowing GitLab to act as a central hub for development with synchronization to other services. The setup is configurable in Settings -> Mirror Repository and can be triggered manually or automatically every hour. An experimental chain of three repositories (Bitbucket -> GitLab -> GitHub) was demonstrated to showcase the functionality.
  14. GitLab.com outage on 2015-09-01This post details a specific incident of GitLab.com being offline for an hour due to perceived filesystem corruption on an NFS server. It describes the initial misdiagnosis based on `dmesg` output, the decision to take the site offline for `fsck`, and the subsequent realization that the errors were recurring daily and not the cause of the immediate slowdown. The post highlights the challenges of diagnosing NFS issues, the decision to abort `fsck` and bring the site back online, and the ongoing effort to migrate data off the problematic filesystem. It also reflects on lessons learned regarding communication during crises and the need for better status updates.
  15. GitLab.com outage on 2015-05-29This post details a specific outage on GitLab.com due to an infrastructure upgrade that moved to a clustered setup. The root cause was identified as a backup script hanging during the filesystem freeze/unfreeze process on the backend server, which prevented workers from mounting the NFS share. The extended downtime was attributed to insufficient training and documentation for on-call engineers regarding the new infrastructure. Next steps included removing the filesystem freeze from the backup script, implementing a secondary backup strategy for SQL data, and initiating regular operations drills for on-call engineers.
  16. Responding to GhostThis post details the process of applying OS-level security patches (glibc vulnerability) to production systems with minimal downtime by leveraging the existing DRBD-based clustered setup for fast failovers. It outlines the steps of updating OS packages and rebooting servers, and specifically highlights how the DRBD-based clustered setup enabled a failover in under a minute to apply updates to the standby server, minimizing downtime for gitlab.com.
  17. WANdisco integrates GitLab Enterprise in Git MultiSiteThis post announces a partnership with WANdisco to integrate Git MultiSite with GitLab Enterprise Edition. This integration leverages WANdisco's active-active replication and Distributed Coordination Engine (DConE) to provide 100% uptime and LAN-speed performance for distributed development teams by turning servers into writable Git peer nodes and resolving conflicts transparently.
  18. GitLab High Availability SolutionsThis post provides an overview of GitLab High Availability Solutions, discussing the trade-offs between increasing uptime and accepting complexity/cost. It structures the discussion starting with the simplest solution (server back-up) and gradually adds levels of complexity, including manual and automated snapshots, dedicated servers, slaves, and clustered/master-master filestore configurations. It emphasizes that there is no one-size-fits-all solution and that the page is a starting point for considering options.