How we built a lab to evaluate data agents

What this post added

This post details the evolution of the 'Evals' system into a comprehensive lab bench called 'The Shoebox' for agent observability and evaluation. It introduces a synthetic business, 'Shorelane Commerce,' with realistic data and complexities to serve as a benchmark for agent evaluations. The post elaborates on the infrastructure for running evaluations as pairwise experiments, the use of rubrics and ground truths, and the creation of hypothesis objective rubrics for targeted evaluations. It also discusses the challenges and maintenance burden associated with such a system.