TechPrufer
Get started
Harvey
Harvey logoHarvey · Feature trail

Review Algorithm

21 posts · Sep 2024 – Sep 2026

What this trail covers

Harvey's Review Algorithm continues to evolve, focusing on enhancing AI evaluation through a multi-pronged approach. This includes leveraging domain expert feedback for curated retrieval datasets and side-by-side LLM comparisons, alongside developing automated evaluation pipelines for continuous monitoring and regression testing. The system now incorporates advanced metrics like precision, recall, and NDCG for retrieval, and Likert-scale ratings and A/B preference tests for answer quality. This post details the application of these evolving AI capabilities within the legal discovery process, specifically focusing on relevance determination and privilege review, and outlines the critical elements of a defensible AI discovery protocol. It highlights the tradeoffs between traditional TAR and generative AI review, emphasizing the importance of rigorous validation methodologies, audit trails, and transparent meet-and-confer disclosures for court acceptance.

All Harvey trails
  1. How Legal Coordinates Compliance Rules Into Action | HarveyThis post details how Harvey's AI capabilities are applied to the compliance workflow, specifically focusing on coordinating compliance rules across regulations, contracts, and policies. It describes the process of applicability assessment, obligation extraction from regulations and contracts, and gap analysis between existing policies and new regulations. The post highlights how AI facilitates tracing the impact of regulatory changes across various business documents and processes, and how this analysis is turned into actionable decisions for the business. It also mentions the integration with Vault for contract-side work and workflow agents for automating processes like obligation mapping and gap analysis.
  2. How Harvey Scales Redline Review to What Really MattersThis post details the technical advancements in Harvey's redline review capabilities, specifically focusing on how AI-assisted comparison scales the process to identify risk-relevant changes. It introduces a materiality framework to test edits for changes in obligations, risk allocation, or deviations from playbook ranges. The system then performs playbook-grounded comparison to surface deviations from approved standards or acceptable fallbacks, presenting them as a prioritized issues list. The output is structured to provide lawyers with plain-text summaries of new obligations, categorization by type, flagging of conflicts, and an open issues list, enabling decision-making within the lawyer's existing working environment.
  3. The Legal Discovery Process: From Review to AnalysisThis post details how AI is shifting the legal discovery process from document review to analysis. It explains the six stages of legal discovery and how AI-assisted review, by enabling the review of entire document productions overnight and providing explanations for classifications, allows legal teams to recover significant time. This recovered time is then redirected towards higher-value activities like case strategy and legal judgment. The post emphasizes the importance of defensibility and traceability in AI-assisted review, highlighting that Harvey's design grounds answers in sources and keeps reasoning open to inspection for verification against underlying documents. It also outlines how to run document review with AI by defining standards, running first-pass review, verifying decisions, and maintaining records for auditability.
  4. What to Look For in Legal Document Comparison SoftwareThis post details enhancements to the review algorithm, focusing on its ability to compare legal documents against an organization's standard positions and precedent. It emphasizes the importance of understanding the legal meaning of changes, not just textual differences. The post highlights how the system grounds comparisons in verifiable source language and integrates with existing workflows like Word and document management systems. It also discusses the capability for bulk document comparison across entire sets and the importance of traceability for defensible output.
  5. Contract Redlining Fundamentals for Legal TeamsThis post details enhancements to the review algorithm, specifically focusing on improving its ability to detect subtle but critical errors in contract redlines. It introduces mechanisms to compare returned drafts against sent versions to surface untracked edits, definition drift, broken cross-references, and deletions that might be misinterpreted as agreement. This is achieved by leveraging AI agents to perform detailed document comparisons, ensuring that the system can reliably identify changes that tracked changes might miss.
  7. Training Frontier Review Table Models With Applied ComputeThis post details the training of a specialized model for Harvey's Review Table product in partnership with Applied Compute. Key contributions include: the development of a synthetic dataset approximating real-world legal document analysis tasks, a novel reward function design that balances answer correctness, schema adherence, and citation quality (precision and recall), and the empirical results showing the custom-trained model's superior performance (0.903 Answer Score, 84.1% citation precision, 91.93% citation recall) and cost reduction (54.8% lower cost per cell) compared to frontier models. It also introduces an experiment comparing single-turn semantic retrieval with a multi-turn agentic retrieval harness for evidence selection.
  8. How to Choose The Best Contract Redlining SoftwareThis post details the technical considerations for ensuring fidelity of marked-up documents when exporting from and importing into the redlining software, specifically focusing on how tracked changes, comments, automatic numbering, and defined terms are preserved across different environments. It highlights the importance of testing this fidelity with plain word processors and the counterparty's tools, and explains how Harvey's integration with Microsoft Word addresses these challenges by maintaining native tracked changes and comments for seamless exchange. It also touches on the procedural constraint of counterparty security reviews and how Harvey's file-based approach mitigates this.
  9. How Real Estate Lawyers Use AI for Lease Review | HarveyThis post details how the review algorithm is applied to real estate lease review and closing. It explains how AI extracts recurring terms, compares documents, identifies inconsistencies, retrieves precedents, builds structured issue lists, and generates first drafts. Specifically, it highlights the application of AI for lease review and abstraction by using repeatable questions, information extraction, classification, summarization, comparison, and reformatting. It also details how AI supports portfolio-level analysis, playbook comparison, and the extraction of obligations, deadlines, and responsible parties for closing deals.
  10. How Legal Teams are Using AI for Legal DiscoveryThis post details how Harvey's AI capabilities are applied to legal discovery, focusing on relevance determination and privilege review. It contrasts traditional TAR with generative AI review, discussing their respective defensibility, explainability, and scalability tradeoffs. The post also outlines a five-element defensible AI discovery protocol (written ESI protocol, validation methodology, sampling-based quality control, audit trail, and meet-and-confer disclosure) crucial for court acceptance. It emphasizes that defensibility is a documentation and process problem, not solely a technology one.
  11. AI for eDiscovery: Faster Document Review Without the RiskThis post details how AI, specifically machine learning, NLP, and GenAI, is being applied to eDiscovery to accelerate document review, privilege analysis, and early case assessment. It highlights the limitations of traditional methods and how AI addresses them by providing consistency, context, and prioritization. Key contributions include explaining conceptual search, sentiment analysis, and entity extraction via NLP, and detailing GenAI's role in early case assessment, document summarization, privilege log drafting, deposition preparation, and narrative timeline generation. The post also emphasizes risk mitigation strategies such as grounding AI outputs in source material, ensuring proportionality and validation through documented protocols, and maintaining privilege and confidentiality through robust data governance controls. It outlines essential features for an eDiscovery AI platform, including data governance, validation support, customization, workflow integration, and purpose-built legal review capabilities.
  12. How to Automate Contract RedliningThis post details a six-step workflow redesign for automating contract redlining, focusing on codifying negotiating positions, configuring intake and classification, deploying AI-assisted first-pass review, implementing risk-tiered routing, reserving human review for critical work, and iterating on the process. It emphasizes that successful automation is a workflow redesign rather than a technology purchase, requiring structured documentation of firm-specific negotiating standards (preferred language, acceptable alternatives, walkaway positions) as the foundation. The post outlines how AI compares incoming contract language against these standards, flags deviations, proposes alternatives with rationales, and routes issues by severity. It also discusses the shift in lawyer roles from first-pass reviewers to strategic editors and the importance of making AI's reasoning visible for associate development. The ROI is tied to establishing a baseline and measuring gains from this redesigned workflow.
  13. How to Redline a Contract With AIThis post details the operational workflow for AI-assisted contract redlining, focusing on the practical steps legal teams take to configure and utilize the system. It introduces the concept of AI-generated tracked-change markups with explanations, and emphasizes the importance of configuring review standards (clause-level rules, escalation criteria, practice-specific customization) as the primary determinant of AI output quality. The post outlines a six-step process: contract intake, configuring review standards, AI-assisted first pass, human review and strategic decisions, iterative back-and-forth with the AI, and final review. It also addresses accuracy and governance, highlighting factual correctness and party-awareness, and the need for a feedback loop to update review standards based on lawyer decisions.
  14. Rebuilding Harvey's Review Algorithm | HarveyThis post details the rebuilding of Harvey's review algorithm for review tables. Key changes include: 1) Reconciling the 'summary' and 'additional context' fields into a single 'answer' field, and introducing a 'reasoning' field to explain the model's interpretation and conclusion. 2) Rethinking the citation methodology to move from cell-level to sentence-level citations for greater granularity and verifiability. 3) Illuminating model reasoning by providing the analytical picture behind the answer. The post also outlines the evaluation process, which involved ALRs generating datasets to measure quality (via preference testing) and latency, and reports a four-fold increase in user preference for the new algorithm, particularly on complex document types.
  15. Contract Intelligence: A Scaled Benchmark for Measuring Contract UnderstandingThis post introduces a new Contract Intelligence benchmark designed to measure LLM contract understanding and extraction accuracy. It defines three types of contract understanding questions (Extraction, Single-Term Review, Multi-Term Understanding) and details the methodology for building the benchmark, including consulting internal lawyers and customers, gathering diverse contracts, and using contracted transactional attorneys and Review tables for responses. The benchmark's ground truth is established by resolving disagreements with Applied Legal Researchers. The post presents findings that lawyers working with Review tables outperform human lawyers or LLMs alone by 5% or more, and analyzes distinct failure modes of LLMs and humans in contract review. It also outlines future plans to expand the benchmark.
  16. Introducing Harvey's New Reasoning CapabilitiesIntroduces 'agentic search' which performs iterative, agent-driven searches over documents to refine its approach and extract all salient insights. This allows for more complete and reliable answers by handling variations in language and overcoming context limits. It also enables flexible queries with mid-thread context additions and broader scope across individual sources and connected knowledge bases. Additionally, 'deep analysis' is now available across any source, combining public and proprietary data for complex, multi-source legal analysis and reports, planning research strategies, and resurfacing with citation-backed reports.
  17. Scaling AI Evaluation Through ExpertiseThis post details Harvey's strategy for scaling AI evaluation by combining expert-led reviews with automated pipelines. It introduces the use of domain expert-curated retrieval datasets and side-by-side LLM comparisons (A/B preference tests, Likert-scale ratings) to assess answer quality, citing a specific example of GPT-4.1 outperforming GPT-4o. It also outlines automated evaluation pipelines for routine checks, production monitoring, and model vetting, including a specialized system for knowledge source identification and citation verification. Key metrics like precision, recall, and NDCG are discussed for retrieval evaluation.
  18. BigLaw Bench: Hallucinations | HarveyThis post details the methodology for defining and measuring hallucinations in LLM outputs, specifically within the context of legal tasks. It introduces a system for breaking down answers into factual claims and verifying them against source documents. The post presents quantitative results comparing Harvey's Assistant model's hallucination rate (0.2%) against foundation models (Claude, ChatGPT, Gemini) on the BigLaw Bench, highlighting Harvey's lower rate despite longer outputs. It also discusses the importance of self-reviewing agents for correcting hallucinations and outlines future plans for hallucination detection, correction, and reasoning transparency.
  19. BigLaw Bench Deep Dive: Sources | HarveyThis post introduces and details the 'source score' as a critical metric for evaluating LLM performance in legal tasks, specifically focusing on the ability of models to accurately cite source documents. It explains the technical challenges in achieving precise source referencing, including disambiguation across multiple documents and the creation of unique token sequences for source demarcation. The post highlights the inherent trade-off between answer quality and source fidelity in LLMs, demonstrating this with examples from ChatGPT and Claude. Harvey's approach to measuring source score for BigLaw Bench is described, which focuses on document-level sourcing. Furthermore, the post frames sourcing as a key modality for intra-model communication in future agentic systems, enabling more reliable information exchange between models.