Compute sample size and significance
A/B Test Design and Analysis
You are tasked with designing and analyzing a two-variant A/B test. The experiment involves equal allocation of users to each variant, and the primary metric is a binary conversion rate.
Experiment Parameters:
- Baseline Conversion Rate (p0): 0.045
- Minimum Detectable Effect (MDE): A +10% relative lift compared to the baseline.
- Statistical Significance (Alpha): 0.05 (two-sided)
- Statistical Power: 0.80
- Assumptions: Independent Bernoulli trials, no continuity correction.
Part A: Sample Size Calculation
Derive the per-variant sample size (n) required for this A/B test. Use the standard normal-approximation formula for two-proportion tests. You must show all intermediate values, including the z-scores for alpha and power, and the pooled variance term, leading to the final per-arm n.
Part B: Impact of Variance Reduction (CUPED)
Assume you apply a variance reduction technique, such as CUPED (Controlled-experiment Using Pre-experiment Data), which effectively reduces the variance of your metric by 30%. Recompute the per-variant sample size required under this new condition. Quantify both the absolute and relative reduction in sample size compared to your result from Part A.
Part C: Mid-Experiment Analysis
Mid-experiment, you observe the following results after collecting data from 50,000 users per variant, without any interim peeking:
- Variant A (Control) Conversion Rate (pA): 0.045
- Variant B (Treatment) Conversion Rate (pB): 0.052
- Sample Size per Variant (nA = nB): 50,000 users
Perform a two-sided z-test for the difference in proportions (pB - pA). Report the calculated z-score, the corresponding p-value, and a 95% confidence interval for the difference in percentage points. Finally, state whether the observed difference is statistically significant at an alpha level of 0.05 and provide a practical interpretation of your findings.
AI interview prep
Core dilemma
Balancing statistical rigor (achieving desired power and significance) with practical constraints on experiment duration, sample size, and resource allocation, while ensuring valid and interpretable results.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Compute sample size and significance
A/B Test Design and Analysis
You are tasked with designing and analyzing a two-variant A/B test. The experiment involves equal allocation of users to each variant, and the primary metric is a binary conversion rate.
Experiment Parameters:
- Baseline Conversion Rate (p0): 0.045
- Minimum Detectable Effect (MDE): A +10% relative lift compared to the baseline.
- Statistical Significance (Alpha): 0.05 (two-sided)
- Statistical Power: 0.80
- Assumptions: Independent Bernoulli trials, no continuity correction.
Part A: Sample Size Calculation
Derive the per-variant sample size (n) required for this A/B test. Use the standard normal-approximation formula for two-proportion tests. You must show all intermediate values, including the z-scores for alpha and power, and the pooled variance term, leading to the final per-arm n.
Part B: Impact of Variance Reduction (CUPED)
Assume you apply a variance reduction technique, such as CUPED (Controlled-experiment Using Pre-experiment Data), which effectively reduces the variance of your metric by 30%. Recompute the per-variant sample size required under this new condition. Quantify both the absolute and relative reduction in sample size compared to your result from Part A.
Part C: Mid-Experiment Analysis
Mid-experiment, you observe the following results after collecting data from 50,000 users per variant, without any interim peeking:
- Variant A (Control) Conversion Rate (pA): 0.045
- Variant B (Treatment) Conversion Rate (pB): 0.052
- Sample Size per Variant (nA = nB): 50,000 users
Perform a two-sided z-test for the difference in proportions (pB - pA). Report the calculated z-score, the corresponding p-value, and a 95% confidence interval for the difference in percentage points. Finally, state whether the observed difference is statistically significant at an alpha level of 0.05 and provide a practical interpretation of your findings.