TechPrufer
Get started
GitHub
GitHub logoGitHub · Feature trail

High Availability Search Architecture for GitHub Enterprise Server

9 posts · Jan 2015 – Mar 2026

What this trail covers

This feature thread tracks the evolution of GitHub's distributed systems for storing and replicating Git repositories, focusing on improving robustness, speed, and flexibility. Initially, GitHub used DRBD for block-level replication, which was sensitive to latency and required replicas to be co-located. The development of DGit (now Spokes) introduced application-level replication, enabling wider geographical distribution of replicas and improved availability through automatic healing and load balancing. This post details the implementation of HAProxy for load balancing MySQL replicas, making the pools context-aware by allowing backend MySQL hosts to signal their inclusion status based on replication lag, server role, and maintenance status. It introduces a graceful failover mechanism using multiple backend pools and ACLs to ensure continuous service availability by switching to a backup pool with lagging replicas when the primary pool has insufficient capacity. The HTTP service for checks is implemented using xinetd and shell scripts, enabling dynamic decision-making without frequent HAProxy configuration changes.

All GitHub trails
  1. How we rebuilt the search architecture for high availability in GitHub Enterprise ServerThis post details the transition from a clustered Elasticsearch setup to using Elasticsearch's Cross Cluster Replication (CCR) for High Availability (HA) GitHub Enterprise Server. It explains the previous challenges with clustered Elasticsearch, such as primary shard movement to replica nodes causing lock states, and the limitations of previous attempts at search mirroring. The new architecture utilizes independent single-node Elasticsearch clusters with CCR, allowing for controlled data replication of durably persisted data. A sample workflow for bootstrapping CCR, including attaching followers to existing indexes and setting up auto-follow policies, is provided. The post also outlines the necessary steps for administrators to adopt this new HA mode, including obtaining a license and configuring `ghe-config app.elasticsearch.ccr true`.
  2. How GitHub Docs’ new search worksThis post details the migration of GitHub Docs site search from an in-memory solution to Elasticsearch. It explains the local Elasticsearch setup for developers, the core search implementation strategy of a single, complex query to Elasticsearch, and the detailed ranking logic. The ranking logic involves a matrix of matching techniques (match_phrase, match with AND/OR) applied to different fields (title, headings, content) with distinct analyzers ('explicit' vs. 'regular' with stemming/synonyms). Each combination has a calculated boost, and the post illustrates the boost values and their impact on result ranking. It also mentions the use of pageview metrics for further ranking refinement.
  3. The technology behind GitHub’s new code searchThis post details the technical implementation of GitHub's new code search engine, 'Blackbird,' built from scratch in Rust. It explains the motivation for building a custom engine over existing solutions, highlighting the unique requirements of code search (e.g., punctuation, no stemming, regex support) and GitHub's scale (200M+ repos, 115TB code). It covers the architecture of the ngram index, the challenges of indexing at scale, and the use of Git's content-addressable hashing and delta encoding for efficient indexing. The post also details the ingest pipeline involving Kafka, crawlers, symbol extractors, and indexer shards, as well as the query path architecture.
  4. MySQL High Availability at GitHubThis post details GitHub's MySQL high availability and primary service discovery solution, which iterates on previous solutions. It moves away from VIP and DNS-based discovery due to its unreliability and slow propagation times, especially for cross-DC failovers. The new solution utilizes orchestrator/raft for detection and failovers, Consul for service discovery (as a KV store), GLB/HAProxy for a proxy layer, and anycast for network routing. This architecture aims to be data center agnostic, tolerant of data center failure, remove unreliable cooperative workflows, reduce total outage time, and achieve lossless failovers.
  5. Stretching SpokesThis post details the engineering effort to enable GitHub's Spokes system to replicate Git repositories across widely separated datacenters. It explains the challenges of high latency impacting Git reference update rates and the optimizations implemented: using three-phase commit with Git transaction work for deterministic updates, incremental checksum computation for efficient state comparison, and prioritizing user-initiated updates over background bookkeeping. The post also highlights the benefits of geo-replication for faster reads and improved disaster recovery.
  7. Building resilience in SpokesThis post introduces the renaming of DGit to Spokes and details the engineering efforts to build resilience in the system. It defines resilience in terms of availability and durability, emphasizing the CAP theorem and Spokes' prioritization of consistency and partition tolerance. The post elaborates on availability by describing how Spokes detects and routes around server and network failures using a combination of heartbeats and real application traffic, employing an MxN failure detection system. For durability, it explains the use of replication (at least three copies in different racks) and a majority-based consensus protocol for writes to ensure data integrity and prevent conflicts, even during concurrent writes or server failures. It also describes the automatic repair process for unhealthy replicas.
  8. Context aware MySQL pools via HAProxyThis post details the implementation of HAProxy for load balancing MySQL read replicas, making the pools context-aware. It introduces a mechanism where backend MySQL servers signal their inclusion status via an HTTP check (responding with 200 for healthy, 503 for lagging/broken). This allows for dynamic exclusion of lagging replicas and servers in maintenance. A graceful failover strategy is implemented using two HAProxy backend pools (main and backup) and ACLs to switch to the backup pool when the main pool has insufficient capacity, ensuring service availability even with stale data. The HTTP check service is implemented using xinetd and shell scripts.
  9. Introducing DGitThis post details the introduction of DGit (Distributed Git), a new application-layer replication system for storing and serving Git content. DGit keeps three copies of every repository on different servers, ensuring high availability even if one or two servers fail. It leverages Git's own protocols for replication and synchronization, storing repositories on local SSDs for performance. The system automatically heals replicas and balances load across a pool of heterogeneous servers, improving resilience and performance compared to the previous DRBD-based system.
  10. How GitHub uses GitHub to document GitHubThis post details the migration of the GitHub Help site from a custom-built Rails application to a static Jekyll site hosted on GitHub Pages. Key technical aspects include: replacing a Rails app with Jekyll for serving documentation, using Jekyll's Collections and data files for content references (conrefs), implementing versioned documentation using Liquid tags and audience flags in `_config.yml`, switching from Elasticsearch to lunr-js for client-side search, and leveraging `html-proofer` and Capybara for testing. The migration aimed to improve performance, reduce operational overhead, and simplify the documentation workflow.