Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust

This post contributes to the field of AI agent evaluation by presenting a comparative study of agentic search (using tools like grep and find) versus vector search for coding agents. It details the methodology for building an evaluation dataset from real-world code fixes (Microsoft's TypeScript Go repo) and scoring agent performance using the repository's test suite. The key finding is that while both approaches achieved comparable accuracy in fixing bugs, vector search was approximately four times more expensive due to its lack of contextual understanding, leading to repeated searches and higher computational costs. The post also touches upon the importance of observability and the 'eval flywheel' for iterative improvement of AI agents.