ATLAS: Evaluating Agents on Search-Intensive Tasks

This post introduces ATLAS, a new benchmark for evaluating agentic search performance. ATLAS focuses on real-world, search-intensive tasks, pairing queries with verified golden answers and an automated pipeline for updates. Key findings include the significant cost-performance trade-off in comprehensive search, the substantial room for improvement in agent completeness, and Exa's position at the cost-performance Pareto frontier. The benchmark design emphasizes requiring search, rewarding search quality and effort, and representing real-world tasks, differentiating it from existing benchmarks by being unmemorized and requiring wide and deep search. The construction pipeline involves frontier agents and multiple search providers to ensure robustness and minimize bias.