Choosing a Sequential Testing Framework — Comparisons and Discussions

This post details Spotify's adoption and reasoning behind using Group Sequential Tests (GST) with alpha spending as the primary framework for its experimentation platform. It contrasts GST with other sequential testing methods like Always Valid Inference (AVI) and Bonferroni corrections, highlighting the pros and cons of each. The decision to use GST is based on its power when maximum sample size can be estimated, and its suitability for both batch and streaming data. The post also explains the problem of 'peeking' and how sequential tests mitigate inflated false positive rates.