Compute p-values for 2 variants vs control
A/B/n Experiment Analysis: P-values, Confidence Intervals, and Ship Decisions
You have conducted an online A/B/n experiment involving one control group and two distinct treatment variants (let's call them Variant 1 and Variant 2).
You are provided with an aggregated dataset, ab_results, structured as follows:
group(STRING): Identifies the experimental group, one ofcontrol,variant_1, orvariant_2.users(INT): The total number of unique users exposed to this specific group.conversions(INT): The count of users within the group who performed the desired binary conversion action.
Assumptions:
- Users were independently assigned to groups, and each user participated in exactly one group.
- The primary metric for evaluation is the conversion rate, calculated as
conversions / users. - The objective is to determine if each treatment variant significantly alters the conversion rate compared to the control group.
- Unless a strong justification for a one-sided test is provided, assume a two-sided hypothesis test is required.
Tasks:
- P-value Calculation: Using Python, calculate the p-value for the following comparisons:
- Variant 1 versus Control
- Variant 2 versus Control Clearly state the chosen statistical test and provide a rationale for its selection.
- Confidence Intervals for Lift: Compute the 95% confidence intervals for the lift (the absolute difference in conversion rates) for each variant when compared against the control group.
- Multiple Testing Correction: Given that two comparisons are being made against the same control group, explain how you would address the issue of multiple testing (e.g., Bonferroni correction, Holm-Bonferroni method, False Discovery Rate (FDR)). Discuss how your chosen method impacts the statistical significance and your ultimate decision.
- Interpretation and Recommendation: Interpret the calculated results in clear, non-technical language. Based on this interpretation, provide a definitive "ship" or "no-ship" recommendation for each variant. Additionally, outline specific conditions or scenarios under which you would recommend running a follow-up experiment.
Important Considerations: Include any critical assumptions or caveats in your analysis, such as considerations for sample size adequacy, potential novelty effects, implications of missing data, or any ambiguities in metric definitions.
AI interview prep
Core dilemma
Making robust, statistically sound business decisions under uncertainty, specifically when comparing multiple experimental groups against a control, while rigorously managing the risk of false positives.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Compute p-values for 2 variants vs control
A/B/n Experiment Analysis: P-values, Confidence Intervals, and Ship Decisions
You have conducted an online A/B/n experiment involving one control group and two distinct treatment variants (let's call them Variant 1 and Variant 2).
You are provided with an aggregated dataset, ab_results, structured as follows:
group(STRING): Identifies the experimental group, one ofcontrol,variant_1, orvariant_2.users(INT): The total number of unique users exposed to this specific group.conversions(INT): The count of users within the group who performed the desired binary conversion action.
Assumptions:
- Users were independently assigned to groups, and each user participated in exactly one group.
- The primary metric for evaluation is the conversion rate, calculated as
conversions / users. - The objective is to determine if each treatment variant significantly alters the conversion rate compared to the control group.
- Unless a strong justification for a one-sided test is provided, assume a two-sided hypothesis test is required.
Tasks:
- P-value Calculation: Using Python, calculate the p-value for the following comparisons:
- Variant 1 versus Control
- Variant 2 versus Control Clearly state the chosen statistical test and provide a rationale for its selection.
- Confidence Intervals for Lift: Compute the 95% confidence intervals for the lift (the absolute difference in conversion rates) for each variant when compared against the control group.
- Multiple Testing Correction: Given that two comparisons are being made against the same control group, explain how you would address the issue of multiple testing (e.g., Bonferroni correction, Holm-Bonferroni method, False Discovery Rate (FDR)). Discuss how your chosen method impacts the statistical significance and your ultimate decision.
- Interpretation and Recommendation: Interpret the calculated results in clear, non-technical language. Based on this interpretation, provide a definitive "ship" or "no-ship" recommendation for each variant. Additionally, outline specific conditions or scenarios under which you would recommend running a follow-up experiment.
Important Considerations: Include any critical assumptions or caveats in your analysis, such as considerations for sample size adequacy, potential novelty effects, implications of missing data, or any ambiguities in metric definitions.