TechPrufer
Get started
GitLab
GitLab logoGitLab · Feature trail

Large Binary Versioning

12 posts · Feb 2015 – Aug 2026

What this trail covers

GitLab now supports Git LFS for versioning large binary files, building upon previous support for Git Annex. The implementation is based on a forked and adapted Git LFS reference server. This post details a tool, 'Git Much Faster,' that benchmarks and optimizes Git clone operations for large repositories, achieving significant reductions in clone times (up to 93%) and disk space usage (up to 98%) through techniques like disabling compression, increasing HTTP post buffers, shallow clones, partial clones with blob filtering, and sparse checkout. It also compares these custom optimizations against Git's Scalar tool, demonstrating superior performance. The optimizations reduce load on the end-to-end stack, including Gitaly Cluster. Use cases span embedded development, enterprise monorepos, media-heavy projects, and CI/CD pipelines. The tool provides Terraform infrastructure for AWS deployment for production-grade testing.

All GitLab trails
  1. Git was built for humans — agents need an upgradeThis post introduces a next-generation source code management (SCM) system designed to address the limitations of traditional Git for agent-scale operations. It details three key problems: the 'clone tax' (excessive data transfer for single file access), 'concurrency collapse' (backend bottlenecks due to high agent load), and 'no isolation' (shared accounts and branch space leading to confusion and lack of auditability). The proposed solution involves a redesigned backend that supports the Git protocol but offers server-side querying for specific data (files, blame, history) without full clones. It features an architecture with an intelligence layer, elastic compute, and object storage, with purpose-built read/write APIs. The system is designed for both cloud and on-premises deployments and has shown significant performance gains in internal testing (up to 50x faster wall-clock time, 2x fewer tokens, 1,000x less network traffic). The post also highlights the importance of provenance and integration with the broader software lifecycle, emphasizing that a standalone Git host is insufficient.
  2. Avoid the massive end-to-end tax of default full history clonesIntroduces the Git Clone Override Policy, a mechanism to automatically enforce optimized Git clone behavior. This policy is implemented via a TOML configuration file stored in the repository and a Go binary that intercepts unqualified `git clone` commands. The policy enforces a 13-step process including tuning `git config` for large transfers, enabling partial clones with `--filter=blob:none`, narrowing fetch refspecs, performing shallow clones with `--depth=1`, and configuring sparse checkout to exclude binary files. This aims to reduce the 'clone tax' exacerbated by agentic AI and other high-frequency cloning scenarios.
  3. GitLab: Built for the agentic engineering eraThis post announces and details the private beta of GitLab's next-generation source code management (SCM) engine, rebuilt for agent-scale concurrency. It addresses the limitations of traditional Git for agent workloads, such as the 'clone tax', concurrency collapse, and lack of isolation. The new SCM runs on the Git protocol for backward compatibility but features a redesigned backend and interfaces built for agents, enabling thousands of agents to experiment safely on any repo. Early internal results show significant improvements in token usage, wall clock time, and network traffic.
  4. Optimize GitLab object storage for scale and performanceThis post details the configuration of GitLab's object storage for scale and performance. It introduces a consolidated form for artifacts, LFS, uploads, and packages, emphasizing the use of IAM roles for authentication and server-side encryption. It also outlines separate configuration for the container registry using the s3_v2 driver and disabling proxy downloads for direct client access. The post provides specific configuration examples for AWS, GCP, and Azure, and recommends using separate buckets for different data types.
  5. Supercharge your Git workflowsThis post introduces 'Git Much Faster,' a benchmarking and optimization script for large Git repositories. It details technical strategies for reducing Git clone times, including disabling `core.compression`, increasing `http.postBuffer`, using shallow clones (`--depth=1`), partial clones (`--filter=blob:none`), and sparse checkout with binary file exclusion. It benchmarks these against standard Git clones and Git Scalar, showing significant performance gains (up to 93% reduction in clone times and 98% reduction in disk space). The post also explains how these optimizations reduce load on server-side components like Gitaly Cluster and provides practical implementation guidance and use cases.
  7. Reduce the load on GitLab Gitaly with bundle URIIntroduces and details the implementation of GitLab's Bundle URI feature for Gitaly. This feature reduces Gitaly load during Git clones by allowing Git clients to pre-download bundled repositories from object storage. The post covers configuration steps for cloud buckets and feature flags, manual and automatic bundle generation methods, and how to enable Bundle URI for terminal clones and within CI/CD pipelines by updating GitLab Runner helper images and enabling `FF_USE_GIT_NATIVE_CLONE`. It also explains the underlying mechanism of how Bundle URI works with Git protocol v2 and contrasts it with the Pack-objects cache alternative.
  8. GitLab's Maven dependency proxy is available in BetaIntroduced the Maven dependency proxy, a new feature that enables enterprises to consolidate on the DevSecOps platform for artifact management. This feature expands the functionality of GitLab's package registry, making pipelines faster and more reliable, and reducing data transfer costs through caching. It allows users to configure an external Java repository, and when a package is requested, GitLab first checks the project and then attempts to pull from the external repository, caching it for future use. The proxy detects changes in the external repository and invalidates cached packages to ensure the latest versions are pulled. It supports authenticated external repositories and is available in the Premium tier.
  9. Future-proofing Git repository maintenanceThis post details the revamp of GitLab's repository maintenance strategy, moving from client-driven RPC calls (Cleanup, RepackIncremental, RepackFull, PackRefs, WriteCommitGraph, GarbageCollect) to a single `OptimizeRepository` RPC. The new strategy analyzes the on-disk state of the repository to perform only necessary maintenance tasks. Key improvements include scaling the number of packfiles based on total packfile size to reduce full repacks for large monorepos, pruning loose objects older than two weeks based on a threshold, compressing loose references when a ratio with the packed-refs file is exceeded, and automatically updating auxiliary data structures like commit-graphs. The rollout on GitLab.com demonstrated a significant reduction in CPU load for repository optimizations.
  10. Speed up your monorepo workflow in GitThis post introduces Git's sparse checkout and partial clone features as methods to speed up workflows in large repositories and monorepos. Sparse checkout allows developers to check out only specific sub-projects within a monorepo, saving local space and reducing `git pull` times. Partial clone, using the `--filter` option, excludes certain types of files (e.g., large blobs) during cloning. These techniques directly address the challenges of large files and deep histories in Git repositories.
  11. 5 Tips for managing monorepos in GitLabThis post provides five practical tips for managing monorepos within GitLab: 1. Utilizing CODEOWNERS files to automate merge request approvals for specific code sections. 2. Employing Git LFS to store large binary files separately, reducing repository size and download times. 3. Leveraging partial clone operations to allow Git to function without a complete repository copy, further optimizing for large repositories. 4. Implementing parent-child pipelines in conjunction with `rules:changes` to run CI jobs only on modified files within a monorepo. 5. Using incremental backups to reduce downtime during repository backups.
  12. How Git Partial Clone lets you fetch only the large file you needThis post introduces Git Partial Clone as a native Git feature that replaces the need for external tools like Git LFS for managing large binary files. It explains how Partial Clone works by allowing users to fetch only the specific blobs they need, rather than the entire repository history. The post details the benefits of Partial Clone, including faster cloning and fetching, reduced disk usage, and improved usability by eliminating the need for separate tools. It also provides a brief history of large file management in Git, discusses network and storage considerations, and outlines performance improvements and usability enhancements.
  13. GitLab no longer requires NFS: The road to Gitaly v1.0This post details the migration from NFS to Gitaly for Git storage on GitLab.com. It describes the challenges of vertical scaling, the decision to move to horizontal scaling using NFS, and the subsequent disadvantages of NFS such as reduced visibility and single points of failure. The introduction of Gitaly, a new Go-based service using Protocol Buffers and gRPC, is explained as the solution to remove NFS dependency. The post highlights the iterative migration process using feature flags, monitoring with Prometheus and ELK, and the extraction of Rails Git code into a reusable library. It also discusses challenges like N+1 problems in the new architecture and the eventual successful unmounting of NFS volumes in production. The next objective is to enable all customers to run without NFS, with administrative tasks slated for v1.1.
  14. Getting started with Git LFSThis post introduces Git Large File Storage (LFS) as a solution for managing large binary files in Git repositories. It explains the problem of bloated repositories caused by committing large files directly and how LFS solves this by storing lightweight pointers in the local repository and downloading the actual files on demand. The post covers installation of LFS on both server (GitLab.com and recent GitLab CE/EE versions) and local machines, how to track files and file patterns using `git lfs track`, and how to view tracked files with `git lfs ls-files`. It emphasizes tracking files as early as possible before they are committed to the repository. The post also highlights the seamless integration of LFS with existing Git workflows and mentions the Tower Git client as a GUI option for managing LFS.
  15. Customer story: A creative agency's GitLab wishlistThis post highlights a customer's need for versioning of design files (Photoshop, InDesign) and GitLab's acknowledgment of this as a 'dream feature'. While not a direct implementation, it indicates future consideration and potential integration with Adobe's solutions. It also touches on the scalability of Docker for GitLab and Mattermost, and GitLab's long-term vision as an end-to-end DevOps platform compared to GitHub's more modular approach.
  16. Feature highlight: Push to a remote repositoryIntroduced the ability to push changes from GitLab to an external repository, complementing the existing functionality to sync changes from a remote repository to GitLab. This allows GitLab to be used as the primary development hub while pushing all changes to another repository, addressing use cases like company policy compliance, maintaining an existing project showcase, or preserving existing integrations. The feature is configurable in Settings -> Mirror Repository and can be triggered manually or automatically hourly. An experimental setup demonstrated a chain of mirroring: Bitbucket -> GitLab -> GitHub, with GitLab triggering builds upon mirror updates.
  17. Announcing Git LFS Support in GitLabIntroduced support for Git Large File Storage (LFS) in GitLab.com, GitLab Community Edition, and GitLab Enterprise Edition. Git LFS uses reference pointers within small text files to point to large files stored on GitLab servers, bypassing previous Git repository size restrictions and improving performance for versioning large assets like video, images, and audio files.
  18. Towards a production quality open source Git LFS serverThis post details GitLab's decision to fork and adapt the Git LFS reference implementation for production use, rather than reusing an existing solution. It outlines the history of large binary versioning solutions (Git Annex, Git Media) and the rationale behind choosing to build upon the Git LFS reference server, including the removal of user management and plans for integration into GitLab CE and EE.
  19. GitLab.com storage limit raised to 10GB per repoGitLab.com storage limit per repository raised from 5GB to 10GB to celebrate the launch of git-lfs. Public and private repositories on GitLab.com remain unlimited with no transfer limit and unlimited collaborators.
  20. GitLab Annex enables versioning of large binaries with gitThis post introduces the integration of git-annex into GitLab Enterprise Edition 7.8, enabling versioning of large binaries without checking their contents directly into Git. It explains how git-annex works by storing symlinks with SHA-1 hashes and how users can add, commit, and sync large files. The integration leverages GitLab Shell for SSH access and provides project-level access control. The post also notes the deprecation and removal of git-annex support in favor of Git LFS in later versions.