How we built a lab to evaluate data agents

This post details the development of 'The Shoebox,' Hex's internal infrastructure for evaluating data agents. It describes the creation of a synthetic business, 'Shorelane Commerce,' with realistic data debt and complexities to serve as a benchmark for agent performance. The Shoebox facilitates pairwise experiment comparisons between candidate and baseline agent runs, supports custom rubrics and ground truths, and integrates with the local Hex dev stack and shared workspaces for consistent evaluation.