Scaling AI Evaluation Through Expertise

This post details Harvey's strategy for scaling AI evaluation by combining expert-led reviews with automated pipelines. It introduces the use of domain expert-curated retrieval datasets and side-by-side LLM comparisons (A/B preference tests, Likert-scale ratings) to assess answer quality, citing a specific example of GPT-4.1 outperforming GPT-4o. It also outlines automated evaluation pipelines for routine checks, production monitoring, and model vetting, including a specialized system for knowledge source identification and citation verification. Key metrics like precision, recall, and NDCG are discussed for retrieval evaluation.