Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering

This post introduces the concept of using LLM evals as a 'verification' step before online experiments ('validation'). It details how LLM evals can assess dimensions like relevance, coherence, and tone at scale, thereby improving the selection of candidates for A/B testing. The post emphasizes the importance of a feedback loop where A/B test results are used to calibrate and improve the LLM evals, ensuring they accurately reflect real user outcomes and business impact. It also highlights the role of guardrail metrics in monitoring secondary outcomes that evals might miss.