How we used DSPy to turn AI evaluations into better responses in Dash chat

This post details the application of DSPy, an open-source framework for optimizing AI systems, to improve the quality of AI evaluations and subsequently enhance the responses of the Dash chat agent. It describes a two-stage process: first, calibrating LLM judges against human-labeled examples using DSPy's optimization algorithms (GEPA, MIPROv2) to align judge scores with human judgment; and second, using these improved judges to optimize the chat agent's system prompt through an automated, evaluation-driven feedback loop. This loop replays candidate prompts on historical chats, scores the outputs, and uses the feedback to propose prompt updates, focusing on concrete failure modes like wrong context selection and incomplete answers.