From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

This post contributes to agent memory and harness engineering by presenting a practical, step-by-step workflow for evaluating and shipping AI agents. It details the use of the Claude Agent SDK, OpenTelemetry, and OpenInference for instrumentation and tracing. Key contributions include: implementing a deterministic ticker check for initial evaluation, developing a custom actionability rubric with explicit criteria and focused evaluators, using human annotations to compare and refine correctness judges, and transforming failing traces into datasets for prompt revision and controlled experiments. The workflow emphasizes reading actual traces to define success and using production monitoring to guide coding agent fixes.