How we optimized Dash's relevance judge with DSPy

This post details the optimization of Dropbox Dash's relevance judge using the DSPy framework. The primary technical contribution is the application of DSPy's GEPA optimizer to adapt a relevance judging model (originally tuned for a proprietary model) to a more cost-effective open-weight model (gpt-oss-120b). This involved defining a clear objective based on human agreement (NMSE) and structural reliability, and using DSPy's reflection loop to generate structured feedback for iterative prompt refinement. The post also outlines the process of measuring human agreement and the challenges of prompt brittleness when migrating between models.