A Framework for Domain-Specific Evaluations

This post introduces a framework for domain-specific LLM evaluations, structured around four pillars: Insights, Research, Approaches, and Context. It details Harvey's expansion of public-facing evaluation work, including the development of new benchmarks for Contract Intelligence and a BLB Challenge focused on difficult legal tasks. The post also discusses the operationalization of evaluations, the incorporation of legal experts, and the importance of providing context to make evaluations meaningful and actionable. It highlights the challenges of current benchmarks and the need for a communal understanding to prevent Goodhart's Law from undermining evaluation effectiveness.