How we used DSPy to turn AI evaluations into better responses in Dash chat

What this post added

This post details the application of DSPy, an AI optimization framework, to improve the Dash chat agent. It describes the process of calibrating LLM judges against human-labeled examples to ensure their evaluations align with human judgment. The post explains how DSPy's optimization algorithms (GEPA, MIPROv2) were used to refine judge prompts and subsequently optimize the chat agent's system prompt. This created a feedback loop where human labels improved judges, judges provided evaluation signals, and those signals enhanced the agent, resulting in fewer incomplete answers and reduced token usage.