Comparing Quantiles at Scale in Online A/B Testing
Mar 1, 2022Mårten Schultzberg (Senior Data Scientist) and Sebastian Ankargren (Data Scientist)
Summary
Introduced a new algorithm for calculating bootstrap confidence intervals for difference-in-quantiles at scale by leveraging the properties of Poisson bootstrap and quantile estimators. This algorithm reduces computational complexity, allowing for inference on hundreds of millions of observations, which was previously intractable. The core innovation lies in analyzing the distribution of ordered indexes that become the sample quantile in Poisson-generated bootstrap samples, approximating it with a Bin(N+1,q) distribution.
More from the Experimentation Platform trail
- Why Spotify Is Not Using Bayesian A/B Testing | Spotify Engineering
- When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering
- Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering
- Why We Use Separate Tech Stacks for Personalization and Experimentation
- Beyond Winning: Spotify’s Experiments with Learning Framework