TechPrufer
Get started
Netflix
Netflix logoNetflix · Feature trail

Experimentation

3 posts · Sep 2021 – Jan 2022

What this trail covers

Netflix is evolving its experimentation capabilities by detailing the statistical concepts of false negatives and statistical power in A/B testing. This post explains how to increase power through effect size, sample size, and reducing metric variability, building upon previous discussions of false positives and statistical significance. The goal is to improve the accuracy and reliability of A/B test interpretations to drive informed product decisions.

All Netflix trails
  1. Interpreting A/B test results: false negatives and powerThis post details the statistical concepts of false negatives and power in A/B testing, explaining their relationship and how to increase power. It covers increasing effect size (bold hypotheses, testing new product areas), sample size (more units, fewer groups), and reducing metric variability. The post uses coin-flipping analogies to illustrate these concepts and their impact on detecting true differences between treatment and control groups.
  2. What is an A/B Test?This post explains the fundamental concept of A/B testing, including random assignment, control vs. treatment groups, and the importance of comparing metrics. It highlights how A/B tests enable causal statements by holding all other factors constant, contrasting this with non-experimental rollout methods. The post also touches on the idea generation process for experiments and the need for careful metric selection, including both hypothesis-specific and general engagement metrics.
  3. Decision Making at NetflixThis post introduces the first in a series on how Netflix uses A/B tests for product decisions, highlighting that experimentation scales by allowing all members to vote with their actions. It frames A/B testing and other causal inference methods as applications of the scientific method for decision-making, forming hypotheses, gathering empirical data, and making conclusions. The post contrasts this with less systematic decision-making paradigms like relying solely on leadership, experts, internal debate, or copying competitors.