Spotify’s New Experimentation Platform (Part 2)

This post details the technical implementation of Spotify's new experimentation platform, focusing on user assignment and analysis. It introduces the 'domain' concept for experiment coordination and the 'holdback' mechanism for evaluating combined experiment effects. A core technical contribution is the 'salt machine,' a system for dynamically re-shuffling users into experiment buckets using a tree of salts and hashing, which allows for starting and stopping experiments without compromising randomization and addresses issues of carryover effects and wasted user space. The post also elaborates on the analysis phase, including the definition of success and guardrail metrics, superiority and non-inferiority tests, sequential and fixed horizon testing, and the sample size calculator which leverages historical data. Validity checks like sample ratio mismatch, pre-exposure activity, crash monitoring, and property collision detection are described. Finally, it outlines the support for gradual rollouts with or without statistical testing.