How do you test two variants vs control?
A/B/n Experiment Analysis: Conversion Rate Optimization
You have conducted an A/B/n experiment involving one control group and two treatment variants (Variant A and Variant B). The primary success metric for this experiment is the conversion rate, defined as whether a user converts or not within the experiment's observation window. Users were independently assigned to one of the three groups.
Experiment Results
The aggregated results are as follows:
| Group | Users (n) | Conversions (x) |
|---|---|---|
| Control | 50,000 | 5,000 |
| Variant A | 50,000 | 5,250 |
| Variant B | 50,000 | 5,400 |
Assumptions
- All statistical tests should be two-sided unless explicitly stated otherwise.
- The significance level (alpha) for hypothesis testing is 0.05.
- The primary metric (conversion rate) is a binomial proportion, and large-sample normal approximations are considered acceptable for calculations.
Tasks
- P-value Calculation: Using Python, compute the p-value for each treatment variant (Variant A vs. Control, and Variant B vs. Control) using an appropriate statistical test for proportions.
- Multiple Comparisons Adjustment: Given the presence of two treatment variants, address the issue of multiple comparisons. Apply a suitable adjustment method (e.g., Bonferroni correction or False Discovery Rate (FDR)) and report the adjusted significance conclusions for each comparison.
- Confidence Intervals: Calculate and provide 95% confidence intervals for the uplift (both absolute and/or relative) for each variant when compared to the control group.
- Recommendation & Next Steps:
- Based on your analysis, provide a clear "ship / no-ship" recommendation for each variant.
- Explain what additional checks you would perform before making a final shipping decision. Consider aspects such as guardrail metrics, segmentation concerns, Sample Ratio Mismatch (SRM) checks, and potential novelty effects.
- Describe how you would effectively communicate the findings and any associated uncertainties to non-technical stakeholders.
You may utilize statistical libraries such as statsmodels or scipy for your computations, or implement the necessary statistical formulas directly.
AI interview prep
Core dilemma
Making robust, statistically sound business decisions from experimental data, balancing statistical rigor (p-values, multiple comparisons, CIs) with practical business considerations (guardrails, communication, uncertainty).
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
How do you test two variants vs control?
A/B/n Experiment Analysis: Conversion Rate Optimization
You have conducted an A/B/n experiment involving one control group and two treatment variants (Variant A and Variant B). The primary success metric for this experiment is the conversion rate, defined as whether a user converts or not within the experiment's observation window. Users were independently assigned to one of the three groups.
Experiment Results
The aggregated results are as follows:
| Group | Users (n) | Conversions (x) |
|---|---|---|
| Control | 50,000 | 5,000 |
| Variant A | 50,000 | 5,250 |
| Variant B | 50,000 | 5,400 |
Assumptions
- All statistical tests should be two-sided unless explicitly stated otherwise.
- The significance level (alpha) for hypothesis testing is 0.05.
- The primary metric (conversion rate) is a binomial proportion, and large-sample normal approximations are considered acceptable for calculations.
Tasks
- P-value Calculation: Using Python, compute the p-value for each treatment variant (Variant A vs. Control, and Variant B vs. Control) using an appropriate statistical test for proportions.
- Multiple Comparisons Adjustment: Given the presence of two treatment variants, address the issue of multiple comparisons. Apply a suitable adjustment method (e.g., Bonferroni correction or False Discovery Rate (FDR)) and report the adjusted significance conclusions for each comparison.
- Confidence Intervals: Calculate and provide 95% confidence intervals for the uplift (both absolute and/or relative) for each variant when compared to the control group.
- Recommendation & Next Steps:
- Based on your analysis, provide a clear "ship / no-ship" recommendation for each variant.
- Explain what additional checks you would perform before making a final shipping decision. Consider aspects such as guardrail metrics, segmentation concerns, Sample Ratio Mismatch (SRM) checks, and potential novelty effects.
- Describe how you would effectively communicate the findings and any associated uncertainties to non-technical stakeholders.
You may utilize statistical libraries such as statsmodels or scipy for your computations, or implement the necessary statistical formulas directly.