Experimentation Platform

19 posts · Sep 2020 – Sep 2026

What this trail covers

Spotify's experimentation platform, Confidence, continues to evolve by introducing the 'fixed-power design.' This design allows experiments to start without a pre-estimated sample size, instead estimating the required sample size from available outcome data and stopping when the current sample size exceeds this estimate. This approach, detailed in Nordin and Schultzberg (2024), enables the use of nonsequential methods for analysis without corrections, ensuring consistent point estimators and asymptotically nominal coverage for confidence intervals, mitigating issues with traditional sequential testing like biased estimators and wasted power.

All Spotify trails

Sep 2026Why Spotify Is Not Using Bayesian A/B Testing | Spotify EngineeringThis post critically analyzes Bayesian A/B testing, arguing that its perceived advantages over frequentist methods are often oversimplified or dependent on specific, complex configurations. It breaks down claims about Bayesian testing regarding peeking, multiple metrics, and the winner's curse, showing how these depend on the chosen prior and stopping rule. The authors advocate for a goal-oriented approach to experimentation, where statistical configurations are chosen based on program objectives rather than a blanket adoption of a particular framework. Spotify's current stance is to leverage existing frequentist tooling, emphasizing that details and specific configurations matter more than the broad label of 'Bayesian' or 'frequentist'.
Aug 2026When Can LLMs Replace Humans in A/B Tests? | Spotify EngineeringThis post introduces the concept of using LLM predictions as surrogates for human outcomes in A/B tests. It formalizes the conditions of 'surrogacy' and 'comparability' required for LLM predictions to validly identify human treatment effects. The post empirically demonstrates that raw LLM predictions are biased, attenuating treatment effects, but that machine learning-based calibration methods (random forest, gradient-boosted trees) can recover the human treatment effect when applied to historical data. It also discusses mitigating LLM sampling temperature noise by averaging multiple predictions. The core limitation identified is that these assumptions are untestable for novel treatments, making human experiments essential for true product innovation. The post suggests LLMs can improve experimental efficiency by filtering weak ideas or reducing variance when new experiments resemble past ones, but not replace human validation by design.
May 2026Better Experiments with LLM Evals — A funnel, not a fork | Spotify EngineeringThis post introduces the concept of using LLM evals as an 'evaluation funnel' before online experiments. It details how LLM judges can verify quality and relevance, thereby raising the hit rate of experiments. A key contribution is the establishment of a feedback loop where LLM evals are run on experiment data to calibrate the judges against real user outcomes. This calibration improves the accuracy of evals and provides diagnostic signals when eval scores and experiment results differ, leading to smarter experiments and more trustworthy evals.
Jan 2026Why We Use Separate Tech Stacks for Personalization and ExperimentationThis post details Spotify's technical and practical reasons for separating personalization and experimentation stacks. It explains that personalization systems, including contextual bandits, reside in the ML stack due to strict infrastructure and performance requirements for model training and low-latency serving. Experimentation systems, like the Confidence platform, are used to evaluate these personalization systems through A/B tests. The post argues against using non-contextual multi-armed bandits for experimentation due to their one-dimensional focus on single metrics and the need for patience with business metrics, advocating for robust A/B testing instead. It emphasizes treating personalization systems as treatment variants within A/B tests and leveraging tight integration between the ML and experimentation platforms.
Sep 2025Beyond Winning: Spotify’s Experiments with Learning FrameworkThis post introduces the Experiments with Learning (EwL) metric as a new way to measure the success of experiments within Spotify's Confidence platform. It defines EwL as experiments that produce valid and decision-ready results, categorizing outcomes into 'Ship', 'Abort due to regression', and 'Powered, no recommendation'. The post details the importance of valid experiments (no sample ratio mismatches, proper metric measurement) and sufficiently powered experiments across all relevant metrics. It also discusses the reasons for experiments not yielding learning, such as failed health checks or insufficient power. The learning rate is highlighted as approximately 64%, emphasizing the value of learning from negative or neutral results.
Advertisement
May 2024Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek atIntroduced the 'fixed-power design' for online experiments, which estimates the required sample size from current outcome data and stops when the current sample size exceeds it. This allows for the use of nonsequential analysis methods without corrections, providing consistent point estimators and asymptotically nominal coverage for confidence intervals, addressing limitations of traditional sequential testing.
Mar 2024Risk-Aware Product Decisions in A/B Tests with Multiple MetricsThis post details the evolution of Spotify's experimentation platform by introducing a risk-aware decision-making engine. It formalizes a decision rule that incorporates four types of metrics: success, guardrail, deterioration, and quality. The post explains how to design and analyze experiments to bound the risks associated with this decision rule, specifically addressing the correction of false negative rates for guardrail and quality metrics, while not adjusting false positive rates for guardrails. It also presents the specific decision rule used at Spotify and in the Confidence platform, emphasizing the importance of aligning experimental design and analysis with this rule to control the risks of incorrect product decisions.
Sep 2023How to Accurately Test Significance with Difference in Difference ModelsThis post introduces and evaluates three methods (averaging, clustering, and permutation testing) for accurately estimating significance in difference-in-differences (DID) models when dealing with autocorrelated time series data. It highlights the limitations of standard OLS significance tests in such scenarios and provides simulation-based evidence on the trade-offs between power and false positives for each method, recommending permutation testing for smaller datasets and clustered standard errors for larger ones.
Aug 2023Coming Soon: Confidence — An Experimentation Platform from SpotifyThis post announces the upcoming commercial release of Spotify's internal experimentation platform, named Confidence. It details the evolution of Spotify's experimentation capabilities over the past decade, from early manual A/B tests with ABBA to the current sophisticated platform. Key technical advancements mentioned include the development of a Metrics Catalog for self-service metric analysis and a coordination engine for managing mutually exclusive experiments and holdback groups. The post also outlines the three ways Confidence will be available to customers: managed service, Backstage plugin, and APIs, highlighting the flexibility and scalability of the platform.
Aug 2023Encouragement Designs and Instrumental Variables for A/B TestingThis post introduces and details the application of encouragement designs and instrumental variables (IV) estimators within Spotify's A/B testing framework. It explains the theoretical underpinnings of these methods, contrasting them with standard A/B tests and tests with one-sided noncompliance. The post elaborates on the four user groups (always-takers, compliers, never-takers, defiers) and how they factor into the IV estimator for calculating the Local Average Treatment Effect (LATE) or Complier Average Causal Effect (CACE). This extends the platform's capabilities to measure feature impact in more challenging experimental setups.
Aug 2023Experimentation at Spotify: Three Lessons for Maximizing Impact in InnovationThis post outlines three strategies for maximizing the impact of experimentation in a mature product organization: 1. Focusing on the specific decision an experiment needs to inform to ensure sufficient, but not excessive, information is gathered. 2. Leveraging localization to innovate for homogeneous user populations, which reduces variance in metrics and increases the chance of detecting significant changes. 3. Breaking down features into their most critical pieces for testing, allowing for earlier, cheaper, and more interpretable results, and mitigating the risk of premature conclusions about product value due to bugs or usability issues in a complete experience.
Jul 2023Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0This post introduces the 'peeking problem 2.0,' which arises in sequential testing with longitudinal data (multiple observations per unit). It details the challenges of analyzing open-ended metrics where within-unit peeking can inflate false positive rates, even with sequential tests. The post outlines the need to separate measurements, metrics, estimands, estimators, and statistical tests, and uses potential outcomes notation to precisely define the statistical goal. It also includes a Monte Carlo simulation study demonstrating the inflated FPRs of standard sequential tests on open-ended metrics.
Jul 2023Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential TestingThis post details the implementation of sequential testing for experiments using longitudinal data. It addresses the 'peeking problem 2.0' by explaining the increased complexity in the dependence structure of test statistics due to multiple observations per unit. The post advocates for a shift in mindset from 'metrics' to 'models', leveraging panel data modeling techniques. It discusses practical considerations like data compression for sufficient statistics and introduces estimators like ROB-OLS, ROB-WLS, and GLS for modeling treatment effects in longitudinal experiments, enabling the use of all available data at all intermittent analyses.
Jun 2023Experimenting at Scale, the Spotify Home WayThis post details the specific tooling and coordination mechanisms used by Spotify's Home surface to run experiments at scale. It introduces Home Config for simplified configuration management by less technical users, highlights the integration of the Experimentation Platform (EP) within Backstage for launching and monitoring, and describes Home QA for debugging and testing. For coordination, it explains the Experiment Tracker for prioritizing experiments and EVA (Experiment Validation Assistant) for automated validation of A/B tests to ensure result validity and reduce misconfigurations. It also mentions the upcoming introduction of Interleaving.
Jun 2023Experimenting with Machine Learning to Target In-App MessagingThis post details the application of uplift modeling and heterogeneous treatment effect prediction to target in-app messaging eligibility. It introduces a multi-headed model inspired by S-learners, T-learners, and Dragonnet, which predicts conditional average treatment effects (CATE) for two different outcomes. The model is trained simultaneously on shared and treatment-specific prediction heads. Offline policy evaluation is used to assess the model's impact on business metrics before A/B testing. The successful rollout resulted in significant improvements in user retention.
Mar 2023Choosing a Sequential Testing Framework — Comparisons and DiscussionsThis post details Spotify's adoption and reasoning behind using Group Sequential Tests (GST) with alpha spending as the primary framework for its experimentation platform. It contrasts GST with other sequential testing methods like Always Valid Inference (AVI) and Bonferroni corrections, highlighting the pros and cons of each. The decision to use GST is based on its power when maximum sample size can be estimated, and its suitability for both batch and streaming data. The post also explains the problem of 'peeking' and how sequential tests mitigate inflated false positive rates.
Mar 2022Comparing Quantiles at Scale in Online A/B TestingIntroduced a new algorithm for calculating bootstrap confidence intervals for difference-in-quantiles at scale by leveraging the properties of Poisson bootstrap and quantile estimators. This algorithm reduces computational complexity, allowing for inference on hundreds of millions of observations, which was previously intractable. The core innovation lies in analyzing the distribution of ordered indexes that become the sample quantile in Poisson-generated bootstrap samples, approximating it with a Bin(N+1,q) distribution.
Feb 2022Search Journey Towards Better Experimentation PracticesThis post details the Search team's journey towards better experimentation practices, complementing the existing experimentation platform. It outlines a three-step roadmap: 1. Individual experiment quality (templates for test specifications, experiment setups), 2. Cross-experiments quality (coordination strategies, guidelines to avoid user exhaustion), and 3. Measure total business impact (quarterly holdbacks for unbiased estimation). It also highlights the importance of 'constant injection of energy' through advocacy and knowledge transfer to embed these practices within teams.
Mar 2021Spotify’s New Experimentation Coordination StrategyThis post details Spotify's migration of its internal experimentation platform to fully rely on 'Bucket Reuse' for experiment coordination. It explains the technical implementation using 1 million buckets, the statistical properties of this approach (including its equivalence to random sampling for difference-in-means estimators and derivation of bias for exclusive experiments), and the benefits of simplicity, scalability, and flexibility. New tools are being developed to help experimenters manage dependencies in exclusive experiment programs.
Nov 2020Spotify’s New Experimentation Platform (Part 2)This post details the technical implementation of Spotify's new experimentation platform, focusing on user assignment and analysis. It introduces the 'domain' concept for experiment coordination and the 'holdback' mechanism for evaluating combined experiment effects. A core technical contribution is the 'salt machine,' a system for dynamically re-shuffling users into experiment buckets using a tree of salts and hashing, which allows for starting and stopping experiments without compromising randomization and addresses issues of carryover effects and wasted user space. The post also elaborates on the analysis phase, including the definition of success and guardrail metrics, superiority and non-inferiority tests, sequential and fixed horizon testing, and the sample size calculator which leverages historical data. Validity checks like sample ratio mismatch, pre-exposure activity, crash monitoring, and property collision detection are described. Finally, it outlines the support for gradual rollouts with or without statistical testing.
Oct 2020Spotify’s New Experimentation Platform (Part 1)This post details the rationale behind replacing Spotify's legacy A/B testing platform (ABBA) with a new Experimentation Platform. It outlines the limitations of ABBA, such as long cycle times for restarts, excessive event volume, insufficient analysis capabilities, and manual coordination of user buckets. The new platform is introduced with three core components: Remote Configuration (using 'properties' instead of feature flags, with PlanOut for policy evaluation and logging 'Config Assigned' and 'Config Applied' events), Metrics Catalog (ingesting raw metric data, joining with experiment group information, aggregating into an OLAP cube, and serving via an API), and Experiment Planner (orchestrating experiments via a Backstage UI, defining treatments, property values, and analysis requirements).