Building Image Understanding for Legal Documents | Harvey

Introduced an on-demand vision system for legal documents that enables AI agents to reason over charts, diagrams, and complex layouts at query time. This system avoids expensive, lossy, and potentially inaccurate indexing-time analysis of all images. Key components include agent-driven invocation, smart page finding via text search, high-fidelity rendering service, and structured visual reasoning with confidence levels. The approach prioritizes text-based analysis first, invoking vision only when necessary. Engineering challenges addressed include scale (narrowing candidate pages to 2-3 in milliseconds), consistency (converting documents to a consistent format and rendering at high DPI), cost (cutting waste by processing only necessary images), and graceful degradation (reporting what can and cannot be determined). Key learnings emphasize that foundation models are effective with text alone, and tool description tuning is critical for agent recall and response quality.