From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

This post contributes to the field of agentic evaluation by introducing the WolfBench framework, which addresses the limitations of single-average scores for agent performance. It details a comprehensive evaluation pipeline that includes six layers: benchmark, runner, sandbox, agent, model service, and settings. The framework emphasizes repeatable runs under uniform conditions to expose reliability gaps, comparing average, best, and worst runs, tasks solved at least once, and tasks solved every time. It also highlights critical considerations such as token usage and total cost, the trade-off between reasoning effort and score, distinguishing infrastructure failures from task failures, and the nuances of zero-temperature runs. The workflow integrates tools like Harbor, Marimo, and Weave for launching, filtering, and inspecting agent runs, providing an interactive leaderboard that visualizes consistency and cost alongside headline scores.