When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering

This post formalizes the conditions under which LLM predictions can serve as valid surrogates for human outcomes in A/B tests, drawing on surrogate endpoint theory. It empirically evaluates LLM-based A/B testing using the Upworthy dataset, demonstrating that raw LLM predictions are biased and attenuate treatment effects. The post identifies two key conditions for LLM outputs to be valid surrogates: surrogacy (LLM output fully mediates the treatment effect) and comparability (the calibration function remains stable). It also highlights that machine learning models are more effective for calibration than linear models and that averaging multiple LLM outputs can mitigate noise. The core limitation is that these assumptions are untestable for future interventions, making human experiments indispensable for novel product innovation. The post concludes that LLM predictions can improve experimental selection and efficiency when new treatments resemble past ones, but substituting them for human outcomes trades identification by design for identification by assumption.