Spotify’s New Experimentation Coordination Strategy
Mar 1, 2021Mårten Schultzberg, Oskar Kjellin, and Johan Rydberg
Summary
This post details Spotify's migration of its internal experimentation platform to fully rely on 'Bucket Reuse' for experiment coordination. It explains the technical implementation using 1 million buckets, the statistical properties of this approach (including its equivalence to random sampling for difference-in-means estimators and derivation of bias for exclusive experiments), and the benefits of simplicity, scalability, and flexibility. New tools are being developed to help experimenters manage dependencies in exclusive experiment programs.
More from the Experimentation Platform trail
- Why Spotify Is Not Using Bayesian A/B Testing | Spotify Engineering
- When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering
- Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering
- Why We Use Separate Tech Stacks for Personalization and Experimentation
- Beyond Winning: Spotify’s Experiments with Learning Framework