Fixing security vulnerabilities with AI

This post details the technical implementation of GitHub's Code Scanning Autofix feature. It explains how LLMs are prompted with vulnerability details and code snippets to generate suggested fixes. The post covers prompt engineering, pre- and post-processing heuristics for selecting code, adding dependencies, specifying code edit formats, and overcoming model errors. It also outlines the evaluation framework used to iterate on the feature, including automated testing of fix validity (removing alerts, no new alerts, no syntax errors, no test failures) and manual triage, which led to a tripled success rate and reduced LLM compute costs.