Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek at
May 1, 2024Mårten Schultzberg, Sr. Manager, Staff Data Scientist
Summary
Introduced the 'fixed-power design' for online experiments, which estimates the required sample size from current outcome data and stops when the current sample size exceeds it. This allows for the use of nonsequential analysis methods without corrections, providing consistent point estimators and asymptotically nominal coverage for confidence intervals, addressing limitations of traditional sequential testing.
More from the Experimentation Platform trail
- Why Spotify Is Not Using Bayesian A/B Testing | Spotify Engineering
- When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering
- Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering
- Why We Use Separate Tech Stacks for Personalization and Experimentation
- Beyond Winning: Spotify’s Experiments with Learning Framework