TechPrufer
Get started
Datadog
Datadog logoDatadog · Feature trail

Git Serving Infrastructure

1 post · Aug 2026

What this trail covers

Datadog rebuilt its Git serving infrastructure with 'gitretriever' to handle a 20x increase in CI traffic. The new system uses independent pods with 'mirrors' to sync with GitHub and 'relays' to serve code to CI workloads. This architecture avoids the replication overhead of previous systems and reduces CPU load by distributing Git pulls across branches and optimizing packfile generation. Key principles include parallel fetching for changed references, serving local copies directly without conse

All Datadog trails
  1. 20× the CI traffic without getting slower: How we rebuilt Git serving at Datadog | DatadogThis post details the evolution of Datadog's Git serving infrastructure with the introduction of 'gitretriever'. The new architecture addresses a 20x increase in CI traffic by moving from a monolithic, replicated Gitaly/Praefect setup to a distributed system of independent 'mirrors' and 'relays'. Mirrors synchronize with GitHub, and relays serve code to CI jobs. This design avoids the CPU and I/O bottlenecks of traditional replication by distributing Git pulls across branches and optimizing packfile generation. Key technical details include parallel fetching for changed references, serving local copies directly, and autoscaling the relay fleet based on load. The system has achieved a 20x traffic growth while maintaining median latency around 40ms and reducing fetch-serving CPU on the previous backend by three to four times.