TechPrufer
Get started
GitHub
GitHub logoGitHub · Feature trail

CodeGen for Language Support

1 post · Aug 2020

What this trail covers

This feature thread tracks the evolution of GitHub's language support system for code analysis. Initially, adding new languages was a difficult and costly process requiring manual grammar writing and complex parsing pipelines. The introduction of CodeGen automates a significant portion of this pipeline, dramatically reducing the cost and effort required to add support for new programming languages, thereby enabling broader code analysis capabilities across GitHub.

All GitHub trails
  1. CodeGen: Semantic's improved language support systemThis post details the introduction of CodeGen, a new language support system for GitHub's Semantic code analysis platform. It addresses the challenges of the previous system, including brittleness, lack of named child nodes in tree-sitter grammars, sub-optimal evaluation machinery with à la carte sum types, and the infeasible time and effort required for manual grammar and assignment writing. The new system introduces modifications to the tree-sitter library to add named child nodes (`field` API), generates a `node-types.json` file from grammars to provide AST structure, and uses Template Haskell to auto-generate language-specific syntax datatypes, significantly reducing manual effort and improving robustness.