Evals in AI: A Deep Dive — Tejas Kumar, IBM

This post contributes to the field of LLM evaluation by demonstrating the limitations of basic assertion-based tests and fuzzy matching for AI systems. It introduces a practical workflow for building and calibrating LLM judges against datasets of customer scenarios, agent responses, and human verdicts. Key contributions include showcasing how to improve judge accuracy by incorporating live policy data (e.g., refund policies via OpenRAG), refining models based on evaluation disagreements, and establishing CI gates with an 80% agreement threshold. The talk also differentiates between pre-deployment checks and runtime harness protections, advocating for deterministic checks before relying on model judgments.