Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust

This workshop details a practical workflow for improving AI agents using production feedback. It starts with a Python support agent built on the OpenAI Agents SDK, adding tracing to establish an offline evaluation baseline and publishing custom scorers. Online automations grade incoming traces, and Topics groups patterns in tasks, sentiment, and workflow issues. A custom facet targets failures missed by default classification. The process involves configuring conversation grouping and sampling, importing sample traces, and inspecting labels and topic maps for investigation. A Loop assistant queries trace data with SQL for analysis. Selected failures are returned to evaluation datasets for checking subsequent changes. A coding agent with the Braintrust CLI and an improvement skill automates repository changes: investigating logs, proposing code and scorer modifications, adding regression examples, and running evaluations. Findings, explanations, and evaluation evidence are packaged into pull requests. The workshop also touches on exposing remote evaluations to a playground for collaborator adjustments.