From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

This post details a practical workflow for evaluating and shipping AI agents that work reliably in production, moving beyond manual trace review to automated problem detection and remediation. It introduces Arize Signal as a layer that groups recurring problems and suggests fixes, which can be translated into GitHub issues, evaluation datasets, new evaluators, or proposed pull requests. The demonstration shows how to add observability to an agent, inspect traces to diagnose issues like missing search filters, and use a coding agent to implement fixes. Regression evaluations are highlighted as a mechanism to prevent introduced bugs, and Signal itself is presented with its own tracing and evaluation suite, enabling a continuous improvement loop.