Why Spotify Is Not Using Bayesian A/B Testing | Spotify Engineering

This post clarifies that Bayesian A/B testing is not a single entity but a family of configurations with different guarantees. It argues against the common oversimplification that Bayesian methods inherently avoid issues like peeking and multiple testing. The post details how specific Bayesian configurations, such as Bayes factor stopping and well-calibrated empirical Bayes priors, are necessary to achieve guarantees like false positive rate control and false discovery rate control, and how these relate to frequentist methods. It also explains that the "winner's curse" is only mitigated by informative priors, not flat priors. Spotify's current stance is that they do not see a need to adopt Bayesian inference due to the complexities and the availability of robust frequentist tooling.