GitLab Workhorse

14 posts · Feb 2013 – Aug 2026

What this trail covers

GitLab Workhorse has evolved from a small Go program addressing Git clone timeouts into a critical component handling a significant portion of HTTP requests to GitLab. It now manages Git HTTP requests, download zip buttons, Git LFS, CI build artifacts, and acts as a smart proxy for all HTTP traffic, simplifying NGINX configurations and improving performance for various features. This includes replacing Gitolite with GitLab Shell for managing Git repositories and SSH authorized keys, reducing complexity. Bundle URI is introduced to offload Git clone load from Gitaly servers by allowing Git to pre-download bundled repositories from object storage before calling Gitaly for remaining objects.

All GitLab trails

Aug 2026Git was built for humans — agents need an upgradeThis post introduces a new architecture for source code management (SCM) designed to handle the scale and concurrency demands of AI agents, addressing issues like the 'clone tax', 'concurrency collapse', and lack of isolation. It details a redesigned backend with purpose-built read/write APIs that allow agents to query specific data server-side instead of cloning entire repositories. The architecture separates an intelligence layer from elastic compute and object storage, supporting both cloud and on-premises deployments. Performance improvements include up to 50x faster wall-clock time, 2x fewer tokens, and 1,000x less network traffic in early testing. It also emphasizes provenance and lifecycle integration as key differentiators beyond just a faster Git backend.
Jul 2026GitLab Transcend Hackathon: What developers built on GitLab OrbitThis post details the outcomes of the GitLab Transcend Hackathon, where developers built solutions on GitLab Orbit. Key technical implementations include agents that analyze blast radius, test impact, and migration costs by querying Orbit's dependency graph. Examples like Sankofa, Carver, and CrossCut showcase how Orbit's context enables features such as identifying downstream callers, pricing migrations, and running only relevant tests. The Contribute Track also saw 61 merge requests merged into the Orbit codebase, adding language support and fixing bugs.
Apr 2026Give your AI agent direct GitLab access with glab CLIThis post introduces the integration of the GitLab CLI (`glab`) with AI agents through the Model Context Protocol (MCP). It details how `glab` provides AI agents with direct, structured access to GitLab projects, enabling them to read issues, comment on merge requests, check pipeline status, and perform other actions programmatically. Key technical aspects include the `glab mcp serve` command to start the MCP server, the automatic addition of `--output json` for structured data, and the use of `glab api` for authenticated access to the full GitLab REST and GraphQL API surface. The post also highlights upcoming improvements like agent-aware help text and machine-readable errors.
Jun 2025Reduce the load on GitLab Gitaly with bundle URIThis post introduces and details the implementation of the Bundle URI feature for Gitaly. It explains how Bundle URI works by allowing Git clients to download pre-bundled repositories from object storage before contacting Gitaly for missing objects, significantly reducing CPU and memory load on Gitaly servers during clones. It provides configuration steps for creating cloud buckets, enabling feature flags (`gitaly_bundle_generation`, `gitaly_bundle_uri`), and generating bundles (manual and auto-generated). It also details how to configure Git clients and GitLab Runners to utilize Bundle URI, including updating helper images and enabling `FF_USE_GIT_NATIVE_CLONE` feature flag. Finally, it contrasts Bundle URI with the Pack-objects cache alternative.
Mar 2025Build and run containers in Remote Development workspacesThis post details the technical steps and configurations required to build and run containers securely within GitLab Workspaces. It covers setting up an AWS EKS Kubernetes cluster, configuring Sysbox as the container runtime, integrating the GitLab agent for Kubernetes and GitLab Workspaces Proxy, enabling sudo access with Sysbox, setting up an NGINX Ingress Controller, and finally, the practical steps for building and running Docker containers within the workspace environment. It provides specific commands and configuration details for each step.
Advertisement
Feb 2024Compose Readers and Writers in Golang applicationsThis post details how GitLab Workhorse utilizes Golang's `io.Reader` and `io.Writer` interfaces and their composite functions (`io.Copy`, `io.MultiWriter`, `io.MultiReader`, `io.TeeReader`) to efficiently handle streaming of large Git data. Specific examples include streaming `InfoRefs` RPC responses via HTTP and implementing the Dependency Proxy by caching upstream images to object storage while simultaneously serving them to users. The post explains the underlying Go interfaces and functions and provides code examples from the GitLab codebase.
Aug 2022Why we implemented our own SSHD solutionThis post details the development and implementation of a custom SSHD solution (`gitlab-sshd`) to replace the previous OpenSSH Server and GitLab Shell architecture for handling Git-over-SSH connections. Key technical contributions include: PROXY protocol support for IP address restriction, graceful shutdown and Kubernetes probe compatibility, Prometheus metrics and profiling capabilities, reduced resource usage via Go goroutines, a reduced attack surface by limiting SSH features, and a simplified architecture. It also covers challenges encountered, such as memory consumption issues, limitations in `golang.org/x/crypto`, and the re-implementation of OpenSSH options, along with lessons learned from incremental rollouts and seeking diverse perspectives.
May 2022How we reduced 502 errors by caring about PID 1 in KubernetesThis post details the debugging and resolution of intermittent 502 errors impacting the GitLab Pages service, specifically caused by GitLab Workhorse failing to shut down gracefully during Kubernetes pod termination. The investigation revealed that Workhorse was not correctly handling the SIGTERM signal, leading to Kubernetes having to forcefully kill the process after the default 30-second grace period. This prolonged the period where the pod was still in the Kubernetes service endpoint but no longer functional, resulting in 502 errors. The fix involved ensuring Workhorse correctly handles SIGTERM and exits within the grace period, preventing the readiness probe failures from causing extended service disruption.
Feb 2022Git fetch performance improvements in 2021, Part 2This post details the optimization of Git fetch performance in 2021, focusing on the implementation of the pack-objects cache and an improved RPC transport for Git HTTP. The pack-objects cache deduplicates identical Git fetch requests, reducing CPU and memory usage. The improved RPC transport uses side channels (Unix sockets or Yamux streams) to bypass gRPC's overhead for bulk data transfer, addressing bottlenecks identified during high fetch traffic scenarios. The pack-objects cache was introduced in GitLab 13.11 and optimized in 14.5. The improved RPC transport for Git HTTP was released in GitLab 14.5.
Nov 2021Deep dive: the tech stack behind SpamcheckThis post details the technical stack and infrastructure choices for the Spamcheck service, a new anti-spam engine. It highlights the use of Golang and gRPC for performance and scalability, and Google Kubernetes Engine (GKE) with Knative for deployment and scaling. GitLab CI/CD is used for automated builds and deployments. The service integrates with the Inspector ML model for spam prediction and utilizes GCP Cloud Monitoring for observability. Performance metrics show Spamcheck surpassing Akismet's false negative and false positive rates. Improvements to block case handling, reCAPTCHA rescue, and shadow-banning are also mentioned.
Jan 2021How we prevented security fixes leaking into our public repositoriesThis post details a critical fix to GitLab's security mirroring process. A design flaw in the mirroring feature caused commits from the private Security repository to be exposed in the public Canonical repository before intended release. The initial workaround involved pausing mirroring during security releases, which caused delays and merge conflicts. The core fix involved switching from Git's `fetch` command to `ls-remote` for branches, preventing Git objects from being persisted on-disk in the public repository. An experiment was conducted to validate this change, followed by a phased rollout using feature flags. A subsequent bug emerged where transient push mirror divergence errors occurred due to `ls-remote` taking too long, causing the local repository to update before remote refs were gathered. This was resolved by swapping the order of reference gathering.
Nov 2020Scaling down: How we shrank image transfers by 93%This post details the implementation of an on-the-fly image scaling solution within GitLab Workhorse. The solution addresses performance and cost issues by resizing images (primarily avatars) to their requested dimensions before delivery to the client. The decision was made to scale dynamically rather than statically due to maintenance overhead and flexibility concerns with existing uploaders. Workhorse was chosen as the integration point over a dedicated proxy like imgproxy due to its existing role as a reverse proxy and alignment with the MVC principle. The implementation involved integrating image scaling logic into Workhorse, leveraging its existing request authentication and business logic validation patterns, and considering approaches like shelling out to an executable or running a sidecar process for the scaling workload.
Mar 2020How to exploit parser differentialsThis post details a security vulnerability in GitLab's Conan package upload functionality. It explains how `gitlab-workhorse` and `gitlab-rails` parse HTTP requests differently, leading to a parser differential. An attacker could exploit this by sending a POST request with an overridden HTTP method (`X-HTTP-METHOD-OVERRIDE: PUT`) to `gitlab-rails`, bypassing `gitlab-workhorse`'s routing and potentially accessing arbitrary files from `/tmp` by manipulating the `file.path` parameter. The vulnerability was patched by implementing request signing in `gitlab-workhorse` and verification in `gitlab-rails`.
Jul 2019What we're doing to fix Gitaly NFS performance regressionsIntroduced a 'cat-file' cache in Gitaly to improve Git IO performance by storing repository data in memory for subsequent requests. This aims to mitigate performance regressions observed with NFS, particularly for large GitLab instances, and address N+1 query issues by reducing repeated Git object read operations. The cache was shipped in GitLab 11.11 behind a feature flag and will be enabled by default in GitLab 12.1. Next steps include automatically enabling Rugged for NFS servers and eventually removing the need for NFS with High Availability for Gitaly.
Sep 2018GitLab no longer requires NFS: The road to Gitaly v1.0This post details the migration from NFS for Git storage to a new service called Gitaly. It describes the challenges of vertical scaling, the disadvantages of NFS (reduced visibility, single point of failure), and the exploration of alternatives like Ceph. The introduction of Gitaly, a Go service using Protocol Buffers and gRPC, is presented as the solution to decouple Git operations and enable horizontal scaling. The post highlights the use of feature flags for gradual rollout, Prometheus and ELK for monitoring, and the extraction of Rails Git code into a vendored library. It also discusses challenges like N+1 problems in the new architecture and the process of unmounting NFS in staging and production environments. The concept of v1.0 is framed as a state of Git infrastructure rather than a code release, with future goals including NFS removal for all customers and the migration of administrative tasks.
Apr 2016A Brief History of GitLab WorkhorseThis post details the evolution of GitLab Workhorse from a weekend project to a critical component. Initially designed to solve Git clone timeouts via HTTP, it expanded to handle 'download zip' buttons, Git LFS, and CI build artifacts. It was renamed from 'gitlab-git-http-server' to 'gitlab-workhorse' due to its expanded functionality. The project then evolved into a smart proxy handling all HTTP traffic to GitLab, simplifying NGINX configurations and improving performance for features like CI artifact browsing and serving raw Git blobs.
Feb 2013GitLab without gitoliteThis post introduces GitLab Shell as a replacement for Gitolite. GitLab Shell is a set of Ruby and shell scripts that manage `/home/git/.ssh/authorized_keys` and `/home/git/repositories`. It consolidates the two system users (gitlab and git) into a single `git` user, simplifies the directory structure, reduces the amount of code related to Gitolite within GitLab, and significantly improves performance by decreasing project creation time by 10x. It also enhances stability by querying GitLab for permissions via API, eliminating sync issues, and simplifies updates via `git pull`.