Three Questions Every Agent Engineer Should Be Able To Answer

This talk introduces a framework for agent engineers to "own their own intelligence" by answering three critical questions: where is the agent failing, how can failures be reliably reproduced, and what needs to be done to fix them. It details techniques for mining production traces using fine-tuned judges for scalable monitoring, creating "world specs" and tasks that mimic production environments (using LangChain's GTM agent as an example), and implementing fixes at both the harness and model layers. The emphasis is on leveraging application-specific insights from real-world usage to improve agent reliability, especially as models and user interactions evolve.