How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases

This post details the engineering approach to building an AI agent (ARIA) that can improve itself. Key technical contributions include: 1. A robust evaluation framework where production traces are converted into offline evaluation tasks, ensuring airtight measurement as benchmarks, evals, and agent configs evolve together. 2. Maintaining byte-for-byte identical production and research agents to ensure accurate comparisons. 3. Defining agent variants in YAML for side-by-side execution and testing. 4. Providing agents with an unconstrained sandbox for experimentation. 5. A scoring mechanism that uses both pass/fail and relative comparison metrics. 6. Generating a large number of evaluation tasks (886 in this case) by treating every production miss and win as a new task, creating a self-improving flywheel. 7. The use of W&B Weave for production and offline tracing to capture detailed execution information. 8. A model-agnostic harness and YAML variants for flexibility in agent configuration and testing.