Git Monorepo Optimization

2 posts · May 2020 – Sep 2025

What this trail covers

Grab has evolved its approach to managing Go modules within a monorepo, addressing challenges with vendoring, private repository access, inconsistent Go versions, and implementing an automated dependency upgrade system (AutoVend Bot). This has improved build integrity, developer productivity, and security by ensuring dependencies are up-to-date. This post details the initial challenges and solutions for adopting Go modules in a multi-module monorepo, focusing on the transition from the 'vendor' directory to Go modules.

All Grab trails

Sep 2025Taming the monorepo beast: Our journey to a leaner, faster GitLab repoThis post details the technical challenges and solutions for optimizing Grab's large Go monorepo. It outlines the scale of the problem, including infrastructure issues like replication lag and developer experience issues like slow clones. The post describes the failure of several standard Git history rewriting tools (git filter-repo, git rebase --onto, patch-based implementation) due to the repository's complexity and scale. The successful solution is a custom, two-phase migration script that reconstructs history by first processing critical tags and LFS objects chronologically in batches, and then processing recent commits. This script addresses memory limits, network usage, and LFS object fetching, enabling a significant reduction in repository size and improvement in performance.
Jun 2023Go module proxy at GrabThis post details the implementation of Athens, a Go module proxy, combined with the `fallback` network mode and the `GOVCS` environment variable, to significantly improve the performance of Go commands (`go get`, `go list`) within Grab's large Go monorepo. It explains how this setup reduces VCS load and speeds up module retrieval from ~18 minutes to ~12 seconds for monorepo modules, while still allowing automatic updates for external modules. The post also quantifies the monorepo size using git-sizer and discusses the consequences of slow Go commands on developer productivity and VCS strain.
Aug 2020Go Modules- A Guide for monorepos (Part 2)This post details the second part of Grab's experience with Go modules in a monorepo, focusing on solutions for catching unexpected `go.mod` changes and addressing dependency issues. It covers implementing CI checks for `go.mod` integrity, workarounds for private repository access and false positives in module checks, the importance of consistent Go versions for vendoring, and the development of the AutoVend Bot for automated dependency upgrades. The post highlights the benefits of Go modules for vendoring, including faster processes and community support, and emphasizes that Go modules require integration with existing workflows rather than being a drop-in solution.
May 2020Go Modules- A Guide for monorepos (Part 1)This post details the initial challenges and solutions for adopting Go modules in a multi-module monorepo, focusing on the transition from the 'vendor' directory to Go modules. It covers the initial obstacles with manual vendoring, the adoption of Glide, and the subsequent switch to `go mod vendor`. Key issues addressed include handling incompatible dependencies, the 'module cannot contain other modules' rule, dependency inheritance, and implicit updates to the `go.mod` file. The post also highlights the utility of `go mod graph` for dependency tracing.
Advertisement