Beyond Winning: Spotify’s Experiments with Learning Framework

This post introduces the Experiments with Learning (EwL) metric as a new way to measure the success of experiments within Spotify's Confidence platform. It defines EwL as experiments that produce valid and decision-ready results, categorizing outcomes into 'Ship', 'Abort due to regression', and 'Powered, no recommendation'. The post details the importance of valid experiments (no sample ratio mismatches, proper metric measurement) and sufficiently powered experiments across all relevant metrics. It also discusses the reasons for experiments not yielding learning, such as failed health checks or insufficient power. The learning rate is highlighted as approximately 64%, emphasizing the value of learning from negative or neutral results.