An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases

This post details the development and demonstration of ARIA, an AI research agent by Weights & Biases, designed to automate AI experiment execution and analysis. It showcases ARIA's ability to launch and monitor training jobs, summarize results, identify patterns across hundreds of experiments, and generate reports and dashboards. The technical architecture involves logging traces to Weave, using LLM judges on live traffic for signal detection, defining tasks as YAML unit tests, and employing a nightly evaluation suite for go/no-go decisions. Key takeaways for productionizing agents include investing in agent-focused observability, treating evals as CI, maintaining human review of traces, and providing context and tools before over-engineering the harness.