The technology behind GitHub’s new code search

This post details the technical implementation of GitHub's new code search engine, 'Blackbird,' built from scratch in Rust. It explains the motivation for building a custom engine over existing solutions, highlighting the unique requirements of code search (e.g., punctuation, no stemming, regex support) and GitHub's scale (200M+ repos, 115TB code). It covers the architecture of the ngram index, the challenges of indexing at scale, and the use of Git's content-addressable hashing and delta encoding for efficient indexing. The post also details the ingest pipeline involving Kafka, crawlers, symbol extractors, and indexer shards, as well as the query path architecture.