BigLaw Bench Deep Dive: Sources | Harvey

This post introduces and details the 'source score' as a critical metric for evaluating LLM performance in legal tasks, specifically focusing on the ability of models to accurately cite source documents. It explains the technical challenges in achieving precise source referencing, including disambiguation across multiple documents and the creation of unique token sequences for source demarcation. The post highlights the inherent trade-off between answer quality and source fidelity in LLMs, demonstrating this with examples from ChatGPT and Claude. Harvey's approach to measuring source score for BigLaw Bench is described, which focuses on document-level sourcing. Furthermore, the post frames sourcing as a key modality for intra-model communication in future agentic systems, enabling more reliable information exchange between models.