How We Benchmark ARIA—and Why It Runs on Weave

This post details the WBAF (W&B Agent Factory) framework for benchmarking ARIA, an agentic system. WBAF ensures production-aligned evaluations by using durable 'Turns' captured from the ARIA service, which include the prompt, history, agent configuration, and environment state. This approach contrasts with simple prompt-based benchmarking by preserving the full context of a production interaction. The framework uses Weave for connected observability and evaluation, allowing for the comparison of current and proposed agent configurations. Tasks are designed to resemble real jobs, preserving page context, conversational history, and external data dependencies. Evaluation uses a variety of scorers (Deterministic, Rubric, Trajectory, Reference) to check specific aspects of agent behavior, such as the existence of created artifacts, the quality of explanations, the evidence used, and adherence to truth. The benchmark suite is managed to remain relevant, focusing on the specific capabilities that have changed, with a broader regression suite available when needed.