GitLab Upgrade and Operationalization

3 posts · Apr 2022 – Sep 2025

What this trail covers

Grab has evolved its GitLab upgrade strategy from a significant backlog to a consistent, efficient routine. This involved detailed preparation, systematic execution of upgrades (including stateful and stateless components), and the development of robust testing and rollback procedures. The team learned to optimize deployment times, leverage version skipping for faster catch-up, and address challenges like Gitaly sync issues, NFS deprecation, and resource right-sizing. This has led to a 50% reduction in upgrade time and enabled catching up on 24 months of upgrades in 11 months.

All Grab trails

Sep 2025Taming the monorepo beast: Our journey to a leaner, faster GitLab repoThis post details Grab's journey to optimize its massive Go monorepo by aggressively trimming its history to improve GitLab's Gitaly replication performance and developer experience. The core problem was identified as replication lag due to the repository's immense size (12.7 million commits, 22.1 million trees, 5.16 million blobs). Traditional Git history rewriting tools like filter-repo, rebase --onto, and patch-based methods failed due to the repository's complexity and scale. The successful solution involved a custom migration script employing a two-phase chronological processing approach: Phase 1 reconstructs history based on critical tags (2,000+ identified) and LFS objects, while Phase 2 integrates recent commits (last month). This tag-based + recent history strategy achieved a 99.9% reduction in commits, reduced replication delays from minutes to seconds, and significantly improved clone times and CI pipeline overhead. The key trade-off was sacrificing deep historical browsing for practical migration feasibility.
May 2024No version left behind: Our epic journey of GitLab upgradesThis post details Grab's journey to establish a consistent GitLab upgrade routine, moving from a 14-month backlog to a streamlined process. It outlines the preparation steps, the 'create before destroy' and 'in-place node rotation' strategies for stateless and stateful components respectively, and the sequential deployment order for Gitaly, Praefect, Sidekiq, App Servers, NFS, and Elasticsearch. Key challenges addressed include Gitaly cluster sync issues, Praefect load distribution, NFS deprecation, handling CI operation surges, and right-sizing resources. The post highlights the adoption of version skipping and the resulting 50% reduction in upgrade time.
Mar 2023Securing GitOps pipelinesThis post introduces Khone, an in-house GitOps solution for managing Terraform-based infrastructure provisioning. Khone addresses limitations of Atlantis by providing enhanced security through custom scripts and workflows, a simplified user experience, and adherence to the DRY principle. It features a three-stage pipeline (Initialization, Terraform, Metric) with security checkpoints like code inspection using python-hcl2, resource validation based on directory structure, and HCL linting. To prevent tampering, it utilizes GitLab's remote pipeline config and centralizes scripts in a separate repository. A development repository (khone-dev) is used for testing pipeline configurations and scripts.
Apr 2022How we reduced our CI YAML files from 1800 lines to 50 linesThis post details the Cauldron Machine Learning (ML) Platform team's implementation of dynamic child pipelines in GitLab CI to manage CI/CD configurations for large repositories. This approach addresses limitations in the number of includes and the size of the main `.gitlab-ci.yml` file by dynamically generating pipeline stages based on detected file changes using a Rust-based utility. The solution involves detecting modified files, extracting relevant information using stop patterns, and templating this into dynamic child pipelines, reducing the main CI YAML file from 1800 lines to 50 lines.
Advertisement