BigLaw Bench: Hallucinations | Harvey

This post details the methodology for defining and measuring hallucinations in LLM outputs, specifically within the context of legal tasks. It introduces a system for breaking down answers into factual claims and verifying them against source documents. The post presents quantitative results comparing Harvey's Assistant model's hallucination rate (0.2%) against foundation models (Claude, ChatGPT, Gemini) on the BigLaw Bench, highlighting Harvey's lower rate despite longer outputs. It also discusses the importance of self-reviewing agents for correcting hallucinations and outlines future plans for hallucination detection, correction, and reasoning transparency.