How We Use ELO Scores to Build Better Legal AI

This post introduces the use of ELO scores, derived from human preference studies, as a primary metric for evaluating and improving AI systems within Harvey. It details how ELO scores are used to understand system performance, set new standards (e.g., for EDGAR search), and iterate on product features (e.g., Vault) by quantifying improvements in preference, accuracy, and latency. Specific examples are provided for knowledge sources (Australia, Austria), EDGAR, and Vault, demonstrating how ELO guides development and differentiates Harvey's AI from generic models and specialized APIs.