Evaluating Agents in Production: Traces, LLM-as-Judge, and Prompt Management at Wonder

This post details how Wonder leverages LangSmith traces, LLM-as-judge evaluators, and automated prompt management to enhance their generative AI application's development and debugging lifecycle. Key contributions include: transitioning from manual log debugging to LangSmith traces for better visibility, implementing LLM-as-judge for evaluating agent responses, automating prompt management to reduce feedback loop times from hours to minutes, and enabling a full-stack bug-fixing workflow where a coding agent can read LangSmith traces, identify code issues, and resolve them autonomously. The post also highlights the empowerment of non-engineers in managing prompts.