Interpret Results and Address Multiple Testing Concerns
Experiment Interpretation with Multiple Testing
Context
An A/B experiment was conducted independently across 30 distinct brands. Within each brand, users were randomly split 50/50 into a Control group and a Test group. For each brand, a hypothesis test was performed using a significance level (α) of 0.05, assuming two-sided tests and independence across all brands.
Question
Upon observing the results, you find:
- 2 brands show a statistically significant lift (positive effect).
- 1 brand shows a statistically significant drop (negative effect).
- 27 brands show no statistically significant difference.
Based on these observations, what conclusions can you draw from these results? Furthermore, how would you account for multiple testing in your answer?
AI interview prep
Core dilemma
Drawing valid statistical conclusions from a large number of simultaneous hypothesis tests while controlling for the increased probability of Type I errors (false positives) and balancing against Type II errors (false negatives).
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Interpret Results and Address Multiple Testing Concerns
Experiment Interpretation with Multiple Testing
Context
An A/B experiment was conducted independently across 30 distinct brands. Within each brand, users were randomly split 50/50 into a Control group and a Test group. For each brand, a hypothesis test was performed using a significance level (α) of 0.05, assuming two-sided tests and independence across all brands.
Question
Upon observing the results, you find:
- 2 brands show a statistically significant lift (positive effect).
- 1 brand shows a statistically significant drop (negative effect).
- 27 brands show no statistically significant difference.
Based on these observations, what conclusions can you draw from these results? Furthermore, how would you account for multiple testing in your answer?