AI-Assisted Code Development and Review

10 posts · Jul 2017 – Sep 2026

What this trail covers

This feature thread tracks the evolution of how AI is integrated into the software development lifecycle. Initially, the focus was on AI's potential to accelerate code writing. This post details improvements to CodeQL's use of custom query packs for enhanced security analysis within GitHub's codebase. It covers the challenges of managing dependencies, publishing query packs to GitHub Container Registry (GCR) for faster iteration, and configuring repositories to leverage these custom packs for improved security auditing and variant analysis.

All GitHub trails

Sep 2026How we make AI coding more cost efficient without sacrificing task qualityThis post details four specific optimizations implemented in GitHub Copilot to improve cost efficiency: 1. Selective output compaction to preserve useful context while reducing repetitive noise in build/test/lint output, while keeping source-like output unchanged. 2. Removal of line-number prefixes from the `view` tool's output, reducing model-inference cost by ~5% in benchmarks and ~3% in online experiments. 3. Prompt compression for the `task`-tool, reducing prompt tokens by ~1300 per turn through meta-prompting and a refined parallelism instruction. 4. Batching completion notifications to deliver background work results directly, eliminating an extra retrieval turn for the agent. These changes were validated through offline benchmarks and controlled online experiments.
Aug 2026Using the GitHub Copilot SDK for JavaThis post introduces the GitHub Copilot SDK for Java, a framework-agnostic client library that allows server-side Java code to programmatically create Copilot agent sessions, register tools using annotations or lambdas, send prompts, and receive structured responses. It highlights the use of virtual threads for efficient agent processing and real-time UI updates via WebSockets. The post details the SDK's features, including `@CopilotTool` and `@CopilotToolParam` for defining tools, `ToolDefinition.from(...)` for inline lambda tools, cross-class tool scanning, customization of system messages, and the `sendAndWait(...)` method for the agentic loop. A sample Jakarta EE 11 application is provided to demonstrate these capabilities.
Aug 2026Turn one giant AI-generated pull request to a reviewable stackThis post introduces and details the concept and implementation of stacked pull requests as a method to manage large AI-generated code changes. It explains the trade-offs of monolithic pull requests versus manually managed chains of smaller PRs. The post introduces the `gh stack` CLI extension and `gh stack skills` for agents to manage these stacks. It outlines a four-layer example of adding product search to a shopping assistant, demonstrating how to decompose work into data, API, chat grounding, and UI layers. It also describes the reviewer's workflow for stacked PRs (read top-down, review bottom-up) and the process of updating and rebasing stacks, highlighting the importance of signed commits and the `gh stack rebase` command.
Jul 2026The cost of saying yes has changedThis post details how AI, specifically GitHub Copilot, can be used to generate a 'first patch' or 'price check' for small feature requests. This allows engineers to evaluate the scope and cost of a change more quickly by examining a concrete diff rather than engaging in lengthy debates. The post emphasizes that the generated code is a probe, not the final product, and that the true cost lies in owning and reviewing the change. It introduces the concept of 'pricing uncertainty' as a new skill for engineers in an AI-assisted world, where the ability to quickly assess the cost of a bounded AI-generated change enables more efficient decision-making.
Jul 2026Better tools made Copilot code review worse. Here's how we actually improved it.This post details how GitHub Copilot code review transitioned from using custom code exploration tools to shared CLI tools (grep, glob, view). Initially, this led to increased review costs and reduced issue detection due to the agent exhibiting broad browsing behavior. The key contribution was rewriting the tool instructions to guide the agent towards a reviewer's workflow: starting from the diff, narrowing the search with grep and glob, and then reading focused ranges with view. This workflow optimization, validated by internal benchmarks, resulted in roughly 20% lower average review cost without compromising quality.
Advertisement
Mar 2026Agent-driven development in Copilot Applied ScienceThis post details the development of `eval-agents`, an agentic system built using Copilot CLI and Claude Opus 4.6, to automate the analysis of coding agent performance against benchmarks. It introduces strategies for agent-driven development, including conversational prompting, architectural refactoring, and a 'blame process, not agents' iteration strategy. The post highlights how these principles enabled rapid development of new agents and skills, emphasizing the importance of clean architecture, documentation, and testing in an agent-first development environment.
Mar 2026Continuous AI for accessibility: How GitHub transforms feedback into inclusionThis post details the implementation of a continuous AI workflow for accessibility feedback. It leverages GitHub Actions, GitHub Copilot, and GitHub Models to automate the process of capturing, analyzing, and prioritizing user feedback. Key contributions include using stored prompts and custom instructions for Copilot analysis, populating over 40 data points automatically, and integrating with project boards for visibility. The workflow also includes a submitter review step guided by AI-generated checklists and a feedback loop for continuous improvement of the AI's behavior.
Mar 2025Finding leaked passwords with AI: How we built Copilot secret scanningThis post details the engineering effort behind Copilot secret scanning for generic passwords. It describes the challenges of using LLMs for security detection, particularly with unconventional file types, and the iterative process of improving offline evaluation frameworks and prompting strategies (including MetaReflection). The post also outlines the development of a workload-aware request management system to scale LLM capacity efficiently and the use of mirror testing to achieve a 94% reduction in false positives, demonstrating increased precision without sacrificing recall.
Feb 2025How GitHub uses CodeQL to secure GitHubThis post details how GitHub Product Security Engineering uses CodeQL for security analysis. It explains the strategy of publishing custom CodeQL query packs to GitHub Container Registry (GCR) for improved deployment and iteration speed compared to embedding queries directly in repositories. The post also covers dependency management for CodeQL query packs, specifically pinning versions of libraries like `ruby-all` to ensure stability, and outlines the release flow for new queries. Finally, it discusses strategies for granting access to these private query packs across multiple repositories within an enterprise.
Feb 2024Fixing security vulnerabilities with AIThis post details the technical implementation of GitHub's Code Scanning Autofix feature. It explains how LLMs are prompted with vulnerability details and code snippets to generate suggested fixes. The post covers prompt engineering, pre- and post-processing heuristics for selecting code, adding dependencies, specifying code edit formats, and overcoming model errors. It also outlines the evaluation framework used to iterate on the feature, including automated testing of fix validity (removing alerts, no new alerts, no syntax errors, no test failures) and manual triage, which led to a tripled success rate and reduced LLM compute costs.
Sep 2019Introducing the CodeSearchNet challengeThis post introduces the CodeSearchNet Challenge, a benchmark and dataset for evaluating code search models. It details the collection of a large corpus of code and documentation from GitHub projects across six languages, the release of baseline models using Transformer architectures, and the creation of an annotated dataset for evaluating code search relevance. The post also describes the data preprocessing pipeline and the methodology for annotating search results.
Oct 2018Applying machine intelligence to GitHub security alertsIntroduces a machine learning model to identify potential security vulnerabilities in package dependencies by analyzing commit messages and diffs. The model filters commits related to security upgrades and identifies packages that may require alerts but lack CVE coverage. Human review of model output is maintained for quality assurance.
Jul 2017Topic Suggestions for Millions of RepositoriesIntroduced 'repo-topix', a topic extraction framework for GitHub repositories. This framework leverages text mining, NLP, and machine learning to suggest descriptive tags. Key technical contributions include: preprocessing READMEs with a heuristics-based tagger, generating candidate topics by eliminating stop words and forming n-grams (limited to size 4 or less), and selecting best topics using a supervised logistic regression model for keyword filtering (trained on 300 examples with features like part-of-speech, n-gram size, and numeric content) and tf-idf based scoring (weighted by phrase frequency and n-gram size, using a large dictionary of IDF scores from millions of READMEs). Also explored canonicalizing topics to address character-level variations and inflections.