Using LLMs to amplify human labeling and improve Dash search relevance

What this post added

This post describes the technical approach to using LLMs to generate relevance labels for Dash search results. It details the RAG pattern used by Dash, the importance of labeled relevance data for training ranking models, and the challenges of traditional human labeling. The core contribution is the hybrid approach of using a small set of human-labeled data to tune LLM prompts and parameters, followed by LLM-generated labels at scale. It also discusses the evaluation of LLM relevance judgments using Mean Squared Error (MSE) and the role of LLMs as teachers for smaller, efficient relevance models.