Legal Agent Benchmark: In-House Contracting | Harvey

This post introduces an extension of the Legal Agent Benchmark (LAB) to specifically evaluate agents' capabilities in contract drafting, review, and negotiation for in-house legal teams. It details the addition of 500 new tasks to LAB, organized by contract types and negotiation stages. The post outlines the agent contracting process within LAB, defining the environment, instruction, output, and verification primitives used to assess an agent's ability to advance contract negotiations. It provides an example of a single contract negotiation task, highlighting the complexity of responding to counterparty redlines and the need for agents to discern between accepting, negotiating, rejecting, and escalating issues. The post also outlines future research directions for improving contract negotiation agents, including expanding task diversity, enabling discovered negotiation states, and developing interactive benchmarks.