How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

This talk details a practical workflow for evaluating AI agents, specifically a financial analyst agent powered by Claude. It demonstrates how to use Arize Phoenix for tracing agent execution, enabling engineers to identify and debug issues like confidently incorrect outputs (e.g., the '116% analyst consensus' bug). The workflow includes writing code-based evaluations, using LLM-as-a-judge for correctness, testing the reliability of these judges, and performing experiments to validate prompt fixes. The process culminates in upgrading the agent model (from Claude Haiku 4.5 to 5.5) based on evidence and gating these evaluations within a CI pipeline using `npm test`. The core contribution is a step-by-step, evidence-driven approach to ensuring AI agent reliability and correctness in production.