20× the CI traffic without getting slower: How we rebuilt Git serving at Datadog | Datadog

This post details the evolution of Datadog's Git serving infrastructure with the introduction of 'gitretriever'. The new architecture addresses a 20x increase in CI traffic by moving from a monolithic, replicated Gitaly/Praefect setup to a distributed system of independent 'mirrors' and 'relays'. Mirrors synchronize with GitHub, and relays serve code to CI jobs. This design avoids the CPU and I/O bottlenecks of traditional replication by distributing Git pulls across branches and optimizing packfile generation. Key technical details include parallel fetching for changed references, serving local copies directly, and autoscaling the relay fleet based on load. The system has achieved a 20x traffic growth while maintaining median latency around 40ms and reducing fetch-serving CPU on the previous backend by three to four times.