Git Performance Monitoring with OpenTelemetry

3 posts · May 2021 – Oct 2023

What this trail covers

This feature thread tracks the evolution of understanding and improving Git performance, particularly for large monorepos. Initially, the focus was on making Git easier to work with in gigantic repositories. This post introduces the adoption of OpenTelemetry for standardized telemetry usage across GitHub, focusing on tracing as the primary entry point for observability. It details the use of OpenTelemetry SDKs and helper libraries to simplify instrumentation, enable automatic correlation of signals (metrics, traces, logs, errors), and improve the developer experience for observability. The team is contributing to the OpenTelemetry project and plans to expand focus to logging and metrics.

All GitHub trails

Oct 2023Measuring Git performance with OpenTelemetryThis post introduces an open-source tool, `trace2receiver`, which acts as an OpenTelemetry receiver component. This component translates Git's Trace2 performance logs into a common format (like OTLP) that can be ingested by OpenTelemetry visualization tools. It demonstrates how this enables distributed tracing of individual Git commands and their child processes, as well as aggregation of performance data over time. Examples show how to use this with SigNoz and Application Insights to analyze `git fetch` and `git status` commands, highlighting the impact of features like FSMonitor and Untracked Cache on performance.
Jun 2022Improve Git monorepo performance with a file system monitorThis post introduces and details the new builtin FSMonitor `git fsmonitor--daemon` feature in Git version 2.37.0. It explains how FSMonitor improves Git performance in large monorepos by reducing the time spent on worktree searches for commands like `git status` and `git add`. The post covers the FSMonitor architecture, including its daemon process, registration with the OS for change notifications, and IPC communication with Git clients. It elaborates on the synchronization mechanism using tokens, PIDs, and SIDs to ensure data integrity and handle potential event drops. The post also provides performance data demonstrating significant speedups on large repositories and explains the different types of files Git considers during worktree searches.
May 2021Why (and how) GitHub is adopting OpenTelemetryThis post details GitHub's adoption of OpenTelemetry to standardize telemetry usage. It highlights the challenges of existing disparate telemetry systems (statsd, syslog, OpenTracing) and introduces OpenTelemetry's OTLP for a common, vendor-neutral format. The post showcases the use of OpenTelemetry SDKs and internal helper libraries for easier instrumentation, including examples of auto-instrumentation for Rails, PG, and ActiveJob. It emphasizes tracing as the main entry point for observability in distributed systems and discusses automatic correlation of signals and the contribution back to the OpenTelemetry project.
Advertisement