AI-Generated Code Is Already Competing With Human Code — Daksh Gupta, Greptile
This talk provides empirical evidence on the quality of AI-generated code in enterprise codebases, analyzing over a million pull requests. It demonstrates that AI-generated code (from models like Codex, Claude Code, and Devin) performs comparably to human-written code across metrics like revert rates, issue severity (P0/P1/P2), and review rounds. Notably, humans were more likely to introduce P0 bugs. The talk also details specific failure modes for different AI models (e.g., SQL injection for Claude, auth bypass for Devin, N+1 queries for Cursor) and discusses the implications for code review at scale, proposing a new validation framework based on user contract adherence, future violation likelihood, and author intent, implemented using blast-radius analysis and sandboxed browser agents.
More from the Agentic Research and Discovery Loops trail
- Build a Research Agent with Claude Managed Agents (Part 2 of 4) — Anthropic
- Scheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — Anthropic
- Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase
- Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI
- From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases