Introducing Exa's Company Search Benchmarks

This post introduces Exa's Company Search Benchmarks, detailing the creation of an ~800 query dataset and an open-sourced evaluation harness. It highlights the challenges of evaluating company search, particularly distinguishing retrieval from LLM memorization, and the need for fresh, structured data. The benchmark includes retrieval and RAG tracks, with specific examples of queries and constraints for attribute filtering, dynamic facts (employee count, funding), and composite queries. It also addresses the temporal nature of dynamic facts and the importance of index freshness. This contributes to the broader goal of building an evaluation ecosystem for entity types and search domains.