Compute A/B significance, CI, and power
A/B Test Statistical Analysis
You have conducted an A/B test over 7 days with the following results:
- Control Group (A): 520 conversions out of 10,000 sessions.
- Variant Group (B): 630 conversions out of 11,500 sessions.
Assume independent Bernoulli trials for conversions.
Tasks
a) Two-Proportion Z-Test: Perform a two-proportion z-test for the null hypothesis H0: pB − pA = 0 against a two-sided alternative. * Compute the pooled-proportion z-statistic. * Compute the two-sided p-value.
b) Confidence Interval: Compute a 95% confidence interval for the difference in proportions (pB − pA) using the unpooled standard error.
c) Sample-Size Planning:
Assuming a baseline proportion p0 = pA (from the A/B test data), determine the equal per-arm sample size n (in sessions) required to detect an absolute lift of 0.7 percentage points (i.e., p1 = p0, p2 = p0 + 0.007). This calculation should be performed at a two-sided α = 0.05 and 80% power, using the normal approximation.
* Use the provided z-values: z0.975 = 1.96 and z0.80 = 0.84.
* Report the formula used and the numeric n rounded up.
d) Multiple Comparisons Correction (Bonferroni): If you were simultaneously running 12 independent metrics in this experiment, what Bonferroni-corrected per-metric α would you use to maintain a family-wise error rate (αFWER) of 0.05?
AI interview prep
Core dilemma
Ensuring the statistical validity and interpretability of A/B test results, balancing theoretical correctness with practical application, and avoiding common pitfalls like misinterpreting p-values or underpowering experiments.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Compute A/B significance, CI, and power
A/B Test Statistical Analysis
You have conducted an A/B test over 7 days with the following results:
- Control Group (A): 520 conversions out of 10,000 sessions.
- Variant Group (B): 630 conversions out of 11,500 sessions.
Assume independent Bernoulli trials for conversions.
Tasks
a) Two-Proportion Z-Test: Perform a two-proportion z-test for the null hypothesis H0: pB − pA = 0 against a two-sided alternative. * Compute the pooled-proportion z-statistic. * Compute the two-sided p-value.
b) Confidence Interval: Compute a 95% confidence interval for the difference in proportions (pB − pA) using the unpooled standard error.
c) Sample-Size Planning:
Assuming a baseline proportion p0 = pA (from the A/B test data), determine the equal per-arm sample size n (in sessions) required to detect an absolute lift of 0.7 percentage points (i.e., p1 = p0, p2 = p0 + 0.007). This calculation should be performed at a two-sided α = 0.05 and 80% power, using the normal approximation.
* Use the provided z-values: z0.975 = 1.96 and z0.80 = 0.84.
* Report the formula used and the numeric n rounded up.
d) Multiple Comparisons Correction (Bonferroni): If you were simultaneously running 12 independent metrics in this experiment, what Bonferroni-corrected per-metric α would you use to maintain a family-wise error rate (αFWER) of 0.05?