Large Binary Versioning

9/10/2025

Introduces 'Git Much Faster', a script for benchmarking and optimizing Git clone operations for large repositories. Details specific optimization techniques including `core.compression=0`, `http.postBuffer=1024M`, shallow clones (`--depth=1`), partial clones (`--filter=blob:none`), sparse checkout for binary exclusion, and compares these against Git's Scalar tool. Provides real-world performance results for Linux kernel, Chromium, and GitLab website repositories, demonstrating significant reductions in clone times and .git directory size. Offers a practical implementation guide and discusses use cases in embedded development, enterprise monorepos, media-heavy projects, and CI/CD pipelines.

9/6/2022

This post introduces and explains two Git features, sparse checkout and partial clone, as methods to speed up monorepo workflows. Sparse checkout allows users to check out only a subset of directories within a repository, reducing local disk space and update times. Partial clone, using the `--filter` option, excludes certain types of files (e.g., large blobs) during the cloning process. Both techniques aim to mitigate the performance issues associated with large repositories and those containing multiple sub-projects.

7/12/2022

This post provides specific tips for managing monorepos in GitLab, focusing on: 1. Using CODEOWNERS for streamlined merge request approvals. 2. Improving Git operation performance with Git LFS for managing large files. 3. Reducing download time with partial clone operations. 4. Leveraging parent-child pipelines for efficient CI/CD on specific file changes. 5. Using incremental backups to eliminate downtime. It also highlights ongoing work in 'Git for enormous repositories', 'Expand SAST scanner support for monorepos', and 'Allow Reports to be Namespace to support monorepos'.

3/13/2020

This post introduces and explains Git's Partial Clone feature, which aims to improve the handling of large binary files in Git repositories. It details how Partial Clone works by allowing Git to fetch only necessary files, thus reducing clone and fetch times and disk usage. The post contrasts Partial Clone with previous solutions like Git LFS and git-annex, highlighting the benefits of a native Git solution. It also provides a practical example of using Partial Clone with the `--filter=blob:none` option and discusses ongoing work related to network and storage optimizations, performance improvements for blob size filtering, and usability enhancements.

1/30/2017

This post introduces and explains Git Large File Storage (LFS) as a solution for managing large binary files in Git repositories. It details the problem of bloated repositories caused by large files, explains how LFS works by storing pointers instead of full files locally, and provides step-by-step instructions for installing LFS on servers and local machines. It also covers how to track files and file patterns using `.gitattributes`, how to get an overview of tracked files, and emphasizes tracking files as early as possible in the repository's lifecycle. The post also highlights the seamless integration of LFS with Git workflows and mentions GUI clients like Tower.

12/15/2016

This post highlights a customer's need for versioning of design files (Photoshop, InDesign) and GitLab's acknowledgment of this as a 'dream feature'. While not a direct implementation, it indicates future consideration and potential integration with Adobe's solutions. It also touches on the scalability of Docker for GitLab and Mattermost, and GitLab's long-term vision as an end-to-end DevOps platform compared to GitHub's more modular approach.

11/23/2015

This post announces the integration of Git Large File Storage (LFS) support into GitLab.com, GitLab Community Edition (CE), and GitLab Enterprise Edition (EE). It explains that Git LFS uses reference pointers to manage large files stored separately on GitLab servers, allowing users to bypass traditional Git repository size limitations (e.g., 1GB) and improve performance. It also mentions that companies using GitLab EE can continue to use git-annex or switch to Git LFS.

8/13/2015

This post details GitLab's decision to fork and improve GitHub's `lfs-test-server` reference implementation to achieve production-ready Git LFS support. It outlines the challenges with large binaries in Git, mentions previous support for Git Annex, and explains the rationale for not reusing GitHub's test server directly due to its user management components. The post also announces the plan to integrate Git LFS support into both GitLab CE and EE, with an estimated alpha release in GitLab 8.1 and beta in 8.2.

4/8/2015

Announced a permanent increase in the storage limit per repository on GitLab.com from 5GB to 10GB.

2/17/2015

This post introduces the integration of git-annex into GitLab Enterprise Edition 7.8, enabling the versioning of large binary files without checking their content directly into Git. It explains how git-annex works by storing symlinks with SHA-1 hashes and how users can add, commit, and sync large files. The integration leverages GitLab Shell for SSH access and provides role-based access control to project files.