GitLab Workhorse

7/20/2026

This post details the outcomes of the GitLab Transcend Hackathon, where developers built solutions on GitLab Orbit. Key technical implementations include agents that analyze blast radius, test impact, and migration costs by querying Orbit's dependency graph. Examples like Sankofa, Carver, and CrossCut showcase how Orbit's context enables features such as identifying downstream callers, pricing migrations, and running only relevant tests. The Contribute Track also saw 61 merge requests merged into the Orbit codebase, adding language support and fixing bugs.

4/27/2026

This post introduces the integration of the GitLab CLI (`glab`) with AI agents through the Model Context Protocol (MCP). It details how `glab` provides AI agents with direct, structured access to GitLab projects, enabling them to read issues, comment on merge requests, check pipeline status, and perform other actions programmatically. Key technical aspects include the `glab mcp serve` command to start the MCP server, the automatic addition of `--output json` for structured data, and the use of `glab api` for authenticated access to the full GitLab REST and GraphQL API surface. The post also highlights upcoming improvements like agent-aware help text and machine-readable errors.

6/24/2025

Introduced and detailed the Bundle URI feature for Gitaly, enabling Git to pre-download bundled repositories from object storage during clones. This significantly reduces load on Gitaly servers, particularly for large monorepos, by offloading the initial download of repository objects. The post outlines configuration steps for cloud buckets, feature flags, manual and auto-generation of bundles, and how to utilize Bundle URI for both terminal and CI/CD pipeline clones, including necessary GitLab Runner helper-image updates and feature flag configurations.

3/3/2025

This post details the technical steps and configurations required to build and run containers securely within GitLab Workspaces. It covers setting up an AWS EKS Kubernetes cluster, configuring Sysbox as the container runtime, integrating the GitLab agent for Kubernetes and GitLab Workspaces Proxy, enabling sudo access with Sysbox, setting up an NGINX Ingress Controller, and finally, the practical steps for building and running Docker containers within the workspace environment. It provides specific commands and configuration details for each step.

2/15/2024

This post details how GitLab Workhorse utilizes Golang's `io.Reader` and `io.Writer` interfaces and their composite functions (`io.Copy`, `io.MultiWriter`, `io.MultiReader`, `io.TeeReader`) to efficiently handle streaming of large Git data. Specific examples include streaming `InfoRefs` RPC responses via HTTP and implementing the Dependency Proxy by caching upstream images to object storage while simultaneously serving them to users. The post explains the underlying Go interfaces and functions and provides code examples from the GitLab codebase.

5/17/2022

This post details the debugging and resolution of intermittent 502 errors impacting the GitLab Pages service, which were traced back to issues with GitLab Workhorse's graceful termination in Kubernetes. The investigation revealed that Workhorse was not cleanly shutting down upon receiving a SIGTERM signal, leading to prolonged periods where it continued to serve 502 errors while the Puma/webservice container was already terminating. This was exacerbated by Kubernetes' default 30-second termination grace period. The solution involved identifying that the Workhorse process was not correctly handling the SIGTERM signal, and ensuring it did so to allow for a faster exit, thus reducing the window for 502 errors. This improved the reliability of the GitLab Pages service during pod lifecycle events.

11/18/2021

This post details the technical stack and infrastructure choices for the Spamcheck service, a new anti-spam engine. It highlights the use of Golang and gRPC for performance and scalability, and Google Kubernetes Engine (GKE) with Knative for deployment and scaling. GitLab CI/CD is used for automated builds and deployments. The service integrates with the Inspector ML model for spam prediction and utilizes GCP Cloud Monitoring for observability. Performance metrics show Spamcheck surpassing Akismet's false negative and false positive rates. Improvements to block case handling, reCAPTCHA rescue, and shadow-banning are also mentioned.

1/4/2021

This post details a critical fix to GitLab's security mirroring process. A design flaw in the mirroring feature caused commits from the private Security repository to be exposed in the public Canonical repository before intended release. The initial workaround involved pausing mirroring during security releases, which caused delays and merge conflicts. The core fix involved switching from Git's `fetch` command to `ls-remote` for branches, preventing Git objects from being persisted on-disk in the public repository. An experiment was conducted to validate this change, followed by a phased rollout using feature flags. A subsequent bug emerged where transient push mirror divergence errors occurred due to `ls-remote` taking too long, causing the local repository to update before remote refs were gathered. This was resolved by swapping the order of reference gathering.

11/2/2020

This post details the integration of an on-the-fly image scaling solution into GitLab Workhorse. The team researched and discarded static scaling due to maintenance overhead and flexibility issues. They opted for dynamic scaling within Workhorse, leveraging its existing infrastructure for request authentication and business logic. A Go-based image scaling executable was developed and integrated as a sidecar process within Workhorse to handle image resizing for avatars and embedded images, achieving a 93% reduction in image transfer size.

3/30/2020

This post details a security vulnerability discovered in GitLab's file upload handling, specifically concerning Conan packages. It explains how a parser differential between gitlab-workhorse and gitlab-rails, combined with Rack::MethodOverride, could be exploited to bypass gitlab-workhorse and access arbitrary files on the server. The post outlines the technical details of the exploit, including the modified PUT/POST requests and the role of `file.path`. The fix involved implementing request signing in gitlab-workhorse and verifying it in gitlab-rails.

7/8/2019

Introduced a 'cat-file' cache in Gitaly to improve Git IO performance by storing repository data in memory for subsequent requests. This aims to mitigate performance regressions observed with NFS, particularly for large GitLab instances, and address N+1 query issues by reducing repeated Git object read operations. The cache was shipped in GitLab 11.11 behind a feature flag and will be enabled by default in GitLab 12.1. Next steps include automatically enabling Rugged for NFS servers and eventually removing the need for NFS with High Availability for Gitaly.

9/12/2018

This post details the migration from NFS for Git storage to a new service called Gitaly. It describes the challenges of vertical scaling, the disadvantages of NFS (reduced visibility, single point of failure), and the exploration of alternatives like Ceph. The introduction of Gitaly, a Go service using Protocol Buffers and gRPC, is presented as the solution to decouple Git operations and enable horizontal scaling. The post highlights the use of feature flags for gradual rollout, Prometheus and ELK for monitoring, and the extraction of Rails Git code into a vendored library. It also discusses challenges like N+1 problems in the new architecture and the process of unmounting NFS in staging and production environments. The concept of v1.0 is framed as a state of Git infrastructure rather than a code release, with future goals including NFS removal for all customers and the migration of administrative tasks.

4/12/2016

This post details the initial development of gitlab-workhorse, starting as a weekend project in Go to address Git clone timeouts with Unicorn. It describes the technical motivations, the integration process via feature flags, and its evolution to handle 'download zip' functionality. The post also covers the renaming to 'gitlab-workhorse' and its expansion to support Git LFS and CI build artifacts, culminating in its role as a smart proxy for all HTTP traffic to GitLab, simplifying NGINX configurations.

2/12/2013

This post details the replacement of Gitolite with a new internal solution called GitLab Shell. The motivation for this change was to address issues with out-of-sync states between GitLab and Gitolite, performance degradation with a large number of repositories, increased system complexity due to two users (gitlab and git), code complexity, and limitations in creating repositories and keys simultaneously. GitLab Shell is described as a set of Ruby and shell scripts for managing `/home/git/.ssh/authorized_keys` and `/home/git/repositories`. The benefits highlighted include a reduction in code complexity, a 10x decrease in project creation time on gitlab.com, improved stability by having GitLab Shell query GitLab for permissions via API instead of maintaining its own ACL, and simpler updates for GitLab Shell.