Risk-Aware Product Decisions in A/B Tests with Multiple Metrics

This post details the evolution of Spotify's experimentation platform by introducing a risk-aware decision-making engine. It formalizes a decision rule that incorporates four types of metrics: success, guardrail, deterioration, and quality. The post explains how to design and analyze experiments to bound the risks associated with this decision rule, specifically addressing the correction of false negative rates for guardrail and quality metrics, while not adjusting false positive rates for guardrails. It also presents the specific decision rule used at Spotify and in the Confidence platform, emphasizing the importance of aligning experimental design and analysis with this rule to control the risks of incorrect product decisions.