Introducing BigLaw Bench to Evaluate LLMs | Harvey

This post introduces BigLaw Bench, a public-facing dataset for evaluating LLMs and model systems on complex legal tasks. It details the methodology for creating benchmark tasks based on lawyer time entries, the evaluation metrics (answer score and source score), and presents initial results showing Harvey's proprietary models outperform leading foundation models. The post also outlines next steps for building a more ambitious benchmark and formalizing industry collaborations.