TechPrufer
Get started
Harvey
Harvey logoHarvey · Feature trail

Legal Agent Benchmark

11 posts · Aug 2024 – Sep 2026

What this trail covers

Harvey's platform continues to evolve its AI agent capabilities, shifting the paradigm from prompt-based assistance to task-oriented execution. The system now empowers agents to autonomously plan, research, execute, and deliver finished work products, with lawyers focusing on goal setting and final review. This evolution spans transactional work, litigation, compliance, and in-house operations, aiming to increase leverage by automating repeatable, high-volume tasks. The platform now integrates advanced capabilities for drafting and verifying legal documents, such as discovery requests, by grounding AI output in case-specific records and ensuring proportionality and compliance with legal rules. This includes features for analyzing pleadings, generating interrogatories, requests for production, and requests for admission, with a focus on traceability and lawyer verification before service. The system also supports redlining, policy gap analysis, and legal data visualization, enhancing efficiency and accuracy in legal workflows.

All Harvey trails
  1. How to Draft Discovery Requests and Responses You Can TrustThis post details how Harvey's AI agents are used to draft discovery requests and responses in litigation. It emphasizes the importance of grounding AI-generated requests in the specific case record, ensuring proportionality under Rule 26(b)(1), and the lawyer's role in verifying and signing these documents. It highlights how Harvey's integration with case documents and its ability to trace requests back to sources facilitate this verification process. The post also discusses the failure modes of generic templates and overbroad requests, and how Harvey helps mitigate these by providing targeted drafts based on the matter's context and governing rules. The capacity gain from AI-assisted drafting and the quality improvement through grounded requests are presented as key benefits.
  2. Harvey Tenet Research PreviewThis post details the development and evaluation of Harvey Tenet, a post-trained open-weight model based on Kimi K3. Key contributions include: 1. Post-training methodology using asynchronous reinforcement learning (RL) with group-sequence policy optimization (GSPO) on a combined corpus of synthetic, public legal, and human expert data. 2. Evaluation demonstrating significant performance improvements on legal benchmarks (LAB, APEX Agents, Redline Bench) and legal knowledge benchmarks (APEX v1, PRBench, LegalBench, CUAD, MAUD). 3. Analysis of cost-efficiency gains through incentivizing efficient tool use and reasoning during RL training. 4. Exploration of specialized capabilities through post-training, including M&A Diligence using Recursive Language Models (RLMs) and Review Tables for structured data extraction, showing improved answer and citation quality at reduced cost.
  3. Extending Legal Agent Bench to M&A Due Diligence | HarveyThis post introduces an extension to the Legal Agent Bench (LAB) specifically for M&A due diligence. It details the creation of synthetic VDR environments containing thousands of documents and millions of tokens to evaluate AI agents' end-to-end legal work capabilities in this complex domain. The contribution focuses on the challenges of context management, exhaustive review, and contextualized judgment required for diligence, and outlines how agents are evaluated against ground truth findings and recommendations.
  4. Training a Legal Agent With Applied Compute | HarveyThis post details the post-training of GLM-5.1 on Harvey's Legal Agent Benchmark (LAB) in collaboration with Applied Compute. The trained model achieved a rubric pass rate of 0.913, outperforming GPT-5.5 xhigh and approaching Opus 4.8 Max. Key technical contributions include: grader alignment analysis using LLM-as-judge with GPT-5 Mini, cost-efficient evaluation by batching rubric criteria, harness optimization with features like compaction and improved tool descriptions, and full-parameter reinforcement learning on Applied Compute Agent Cloud. Model analysis highlights domain-level shifts, reduced tool usage with more precise application, and improvements in artifact completeness, specificity, and grounding.
  5. Legal Agent Benchmark: In-House Contracting | HarveyThis post details the extension of Harvey's Legal Agent Benchmark (LAB) to evaluate AI agents on in-house contracting workflows. It introduces 500 new tasks focused on contract drafting, review, and negotiation, organized by contract types and negotiation stages. The benchmark models the contract negotiation process using primitives like environment, instruction, output, and verification, with all-pass scoring for evaluation. The post outlines the structure of a single contract negotiation task, including the context provided to the agent (contract, playbooks, deal context) and the expected output (redline, issues list). It also discusses future research directions such as increasing task diversity, discovering negotiation states, and building interactive benchmarks for autonomous negotiation.
  7. Post-Training Open Legal Agents With Baseten ResearchThis post details research into post-training open-weight legal agents using the Legal Agent Benchmark (LAB) and a legal-agent harness. Key contributions include: 1. Demonstrating that post-training can significantly improve agent performance on legal tasks, addressing domain expertise gaps. 2. Showing that agents trained against LAB criteria converge on effective research strategies (e.g., dominant 'read' tool over 'grep'). 3. Validating a natural-language compaction strategy that significantly improves the 'all-pass' rate for legal deliverables by managing context at scale. 4. Post-training Qwen3.5-27B using Iterative SFT (iSFT) with a compaction harness, achieving performance within the closed-source frontier band on LAB. The research highlights the interplay between post-training and harness optimization, and explores methods for handling imperfect rollouts during training.
  8. Legal Agent Benchmark Initial Results | HarveyThis post presents initial results from benchmarking frontier AI models on the Legal Agent Benchmark (LAB). Key findings include: 1) Frontier models complete less than 10% of legal tasks end-to-end under a strict all-pass standard, indicating legal work is not yet saturated. 2) Model intelligence is unevenly distributed across legal practice areas, with different models leading in different specializations, demonstrating 'jagged intelligence'. 3) High performance comes at a significant cost, with top models costing ~$50 per task and taking over 20 minutes of latency. The post also analyzes agent behavior, identifying common action sequences (read, search, execute, write, validate, edit) and correlating specific behaviors like thorough research and self-correction with higher performance. This analysis informs the strategy for multi-model agent deployments.
  9. Introducing Harvey’s Legal Agent BenchmarkThis post introduces Harvey's Legal Agent Benchmark (LAB), an open-source benchmark designed to evaluate and improve AI agent capabilities for long-horizon legal work. It details the benchmark's structure, mirroring real-world legal assignments with instructions, client matters, and reviewable work products. The post explains the client-matter-centric task definition, the all-pass grading system for evaluation, and the distribution of tasks across 24 legal practice areas. It also discusses the rationale for building LAB, contrasting it with existing short-horizon benchmarks and highlighting its role in measuring ROI and guiding deployment decisions for AI systems in law firms.
  10. How We Use ELO Scores to Build Better Legal AIThis post introduces the use of ELO scores, derived from human preference studies, as a primary metric for evaluating and improving AI systems within Harvey. It details how ELO scores are used to understand system performance, set new standards (e.g., for EDGAR search), and iterate on product features (e.g., Vault) by quantifying improvements in preference, accuracy, and latency. Specific examples are provided for knowledge sources (Australia, Austria), EDGAR, and Vault, demonstrating how ELO guides development and differentiates Harvey's AI from generic models and specialized APIs.
  11. Expanding BigLaw BenchThis post details the expansion of the BigLaw Bench (BLB) benchmark to include global jurisdictions, core legal practice areas (UK, Australia, Spain, Capital Markets, M&A, Fund Formation, Litigation, Arbitration, IP), and complex legal research tasks. It reflects on learnings from the original BLB, emphasizing task performance as the most accurate measure of model capabilities and discussing how BLB is used to understand general model gaps (e.g., distinguishing evidentiary rules) and test model behaviors (e.g., sycophancy). The post previews methodological deep dives into each new benchmark area and announces upcoming updated BLB scores for major models.
  12. Introducing BigLaw Bench: ArenaThis post introduces BigLaw Bench: Arena (BLB: Arena), an internal system for head-to-head evaluation of AI systems on legal tasks using scaled, pairwise comparison. It details the process of lawyers comparing two outputs, providing preferences and reasoned explanations. The system aggregates these votes into Elo scores for quantitative ranking and analyzes qualitative feedback into preference drivers (Alignment, Trust, Presentation, Intelligence). Recent results are presented comparing Harvey Assistant (built on GPT-5) against prior implementations and leading foundation models, demonstrating Harvey Assistant's superior performance and confirming intelligence as a key preference driver. The post also discusses the limitations of Elo scores and the importance of qualitative analysis.
  13. A Framework for Domain-Specific EvaluationsThis post introduces a framework for domain-specific LLM evaluations, structured around four pillars: Insights, Research, Approaches, and Context. It details Harvey's expansion of public-facing evaluation work, including the development of new benchmarks for Contract Intelligence and a BLB Challenge focused on difficult legal tasks. The post also discusses the operationalization of evaluations, the incorporation of legal experts, and the importance of providing context to make evaluations meaningful and actionable. It highlights the challenges of current benchmarks and the need for a communal understanding to prevent Goodhart's Law from undermining evaluation effectiveness.
  14. Scaling AI Evaluation Through ExpertiseThis post details Harvey's structured evaluation strategy for AI performance, emphasizing three core pillars: expert-led reviews, automated evaluation pipelines, and dedicated data services. It elaborates on expert-curated retrieval datasets, evaluation metrics (precision, recall, NDCG), and answer quality assessment using A/B preference tests and Likert-scale ratings, citing a specific comparison between GPT-4.1 and GPT-4o. The post also outlines the benefits and components of automated evaluation pipelines, including routine evaluations, production monitoring, and model vetting, with a specific example of a Knowledge Source Identification system for citation verification.
  15. BigLaw Bench: Hallucinations | HarveyThis post details Harvey's methodology for defining, measuring, and reducing hallucinations in LLMs. It introduces a system for breaking down answers into factual claims and verifying them against source documents, and presents hallucination rates for Harvey's Assistant model compared to foundation models (Claude, ChatGPT, Gemini) on BigLaw Bench tasks. It also outlines future plans for agents that can proactively correct hallucinations and enhance transparency in model reasoning.
  16. BigLaw Bench Workflows: SPA Deal PointsThis post introduces a new workflow dataset for evaluating legal agents, specifically for extracting deal points from Share Purchase Agreements (SPAs). It details the dataset collection, annotation process, and the reasoning challenges faced by foundation models. Harvey's SPA agents achieve 98.47% accuracy, significantly outperforming general foundation models like GPT-4o and Gemini. The post also discusses the importance of transparent reasoning and sources in AI-generated legal outputs and outlines plans for expanding BigLaw Bench with more workflows.
  17. Introducing BigLaw Bench to Evaluate LLMs | HarveyThis post introduces BigLaw Bench, a public-facing dataset for evaluating LLMs and model systems on complex legal tasks. It details the methodology for creating benchmark tasks based on lawyer time entries, the evaluation metrics (answer score and source score), and presents initial results showing Harvey's proprietary models outperform leading foundation models. The post also outlines next steps for building a more ambitious benchmark and formalizing industry collaborations.