A brief history of code search at GitHub

This post details the historical evolution of code search at GitHub, moving from Solr and git grep to Elasticsearch. It highlights the challenges of searching code due to its unique characteristics (punctuation, scale, rate of change, performance requirements) and explains how GitHub's custom text analysis pipeline for Elasticsearch was designed. Specifically, it details the use of a pattern tokenizer to split documents and extract tokens, including word parts from CamelCase and snake_case identifiers, while ignoring certain punctuation characters. This approach was a result of careful experimentation with different strategies to balance index size, performance, and query capabilities.