Contract Intelligence: A Scaled Benchmark for Measuring Contract Understanding

This post introduces a new Contract Intelligence benchmark designed to measure LLM contract understanding and extraction accuracy. It defines three types of contract understanding questions (Extraction, Single-Term Review, Multi-Term Understanding) and details the methodology for building the benchmark, including consulting internal lawyers and customers, gathering diverse contracts, and using contracted transactional attorneys and Review tables for responses. The benchmark's ground truth is established by resolving disagreements with Applied Legal Researchers. The post presents findings that lawyers working with Review tables outperform human lawyers or LLMs alone by 5% or more, and analyzes distinct failure modes of LLMs and humans in contract review. It also outlines future plans to expand the benchmark.