TechPrufer
Get started
Harvey
Harvey logoHarvey · Feature trail

Retrieval System Benchmarking

2 posts · Jul 2024 – Feb 2025

What this trail covers

Harvey's retrieval system has been benchmarked against common embedding-based and reranking methods using the BigLaw Bench Retrieval dataset. The system demonstrates superior performance, identifying up to 30% more relevant content across diverse legal document types, including contracts and research databases. This is attributed to the incorporation of metadata, task-specific features like recency, and LLM-based reasoning for relevancy judgments. Future work will extend benchmarks to more legal document types and explore custom embedding models. This post details a partnership with Voyage AI to fine-tune an embedding model on US case law, reducing irrelevant material by nearly 25% compared to off-the-shelf models and achieving this with 1/3 of the embedding dimensionality, leading to storage and latency benefits. This custom model, voyage-law-2-harvey, is combined with other proprietary search methods to further improve retrieval accuracy.

All Harvey trails
  1. Enterprise-Grade RAG Systems | HarveyThis post details Harvey's approach to building enterprise-grade RAG systems, focusing on the technical considerations for vector databases. It highlights the challenges of sparse vs. dense representations, performance at scale, accuracy, complex query structures, domain-specific data, and evaluation. The post compares Postgres with PGVector and LanceDB Enterprise, ultimately favoring LanceDB for its strengths in latency, accuracy, ingestion throughput, scalability, data privacy, and hosting. It also emphasizes the importance of domain expertise in refining retrieval systems, citing the collaboration with PwC's tax professionals as an example.
  2. BigLaw Bench Deep Dive: Retrieval | HarveyThis post details the expansion of the BigLaw Bench to include retrieval-focused benchmarks. It presents an evaluation of Harvey's proprietary retrieval system against common embedding-based and reranking methods, demonstrating a 30% improvement in identifying relevant content. The post highlights the unique challenges of legal retrieval (contracts, discovery emails, research databases) and attributes Harvey's superior performance to the incorporation of metadata, task-specific features (like recency), and LLM-based reasoning for relevancy judgments. It also discusses the importance of domain- and task-specific optimization for retrieval efficacy.
  3. BigLaw Bench Deep Dive: Sources | HarveyThis post details the rationale and methodology behind measuring LLM source referencing accuracy, introducing the 'source score' as a key metric in BigLaw Bench. It explains the technical challenges in achieving specific and unique source citations, the trade-offs between answer quality and source fidelity in LLMs, and how Harvey's system outperforms others in this area. The post also frames sourcing as a critical component for future agentic systems, enabling reliable intra-model communication.
  4. Harvey partners with Voyage to build custom legal embeddingsThis post details a partnership with Voyage AI to fine-tune an embedding model on US case law, named voyage-law-2-harvey. This custom model reduces the amount of irrelevant material returned in top results by nearly 25% compared to the next best off-the-shelf embedding models, while using 1/3 of the embedding dimensionality. The training involved both raw case law text and a dataset of exemplar questions with expert annotations. The performance was evaluated using NDCG@10 and Recall@100 metrics.