Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

This post contributes to agent evaluation by highlighting the limitations of LLM-as-judge verifiers on benchmarks like WebVoyager, which can overstate agent success. It introduces the 'Universal Verifier' which uses task-specific rubrics, ranks relevant screenshots as evidence, isolates errors, catches hallucinations, and distinguishes controllable from uncontrollable failures. This approach significantly reduced false positives and improved agreement with human judgments. The research also investigates the use of autoresearch to build verifiers, demonstrating its potential while acknowledging the continued importance of human oversight.