Determine A/B test sample size drivers
A/B Test Sample Size with Variations and Overdispersed Metric Case
You are tasked with planning a two-arm A/B test on a binary conversion metric with the following parameters:
- Baseline conversion (p0): 8.0% (0.08)
- Relative Minimum Detectable Effect (MDE): 8% (This implies p1 = 1.08 × p0 = 0.0864, resulting in an absolute lift Δ = 0.0064)
- Two-sided alpha (α): 0.05
- Statistical power: 80% (β = 0.20)
- Allocation ratio: 1:1 (equal sample sizes per arm)
Assume a z-test for the difference in proportions using a normal approximation.
Tasks
(a) Derive and compute the required per-arm sample size using the normal approximation to the difference in proportions. Clearly show the z-terms you use.
(b) Holding all other parameters constant (relative to part a), qualitatively and quantitatively describe how the required sample size changes when each of the following is modified individually:
- MDE halves.
- α is tightened to 0.01 (two-sided).
- Power is increased to 90%.
- Allocation is 75/25 (treatment/control).
- Observations are clustered at the user level with an Intraclass Correlation Coefficient (ICC) = 0.02 and an average cluster size (m) = 5. Compute the design effect and apply it.
- You adopt group-sequential monitoring with two equally spaced looks using O’Brien-Fleming boundaries.
(c) If the primary metric is an overdispersed count (Negative Binomial 2, NB2), outline how you would re-estimate the required sample size.
AI interview prep
Core dilemma
Balancing statistical rigor and the ability to detect meaningful effects with practical constraints on experiment duration, cost, and complexity of implementation. The core challenge is determining the minimum necessary sample size to make reliable business decisions under various real-world data conditions.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Determine A/B test sample size drivers
A/B Test Sample Size with Variations and Overdispersed Metric Case
You are tasked with planning a two-arm A/B test on a binary conversion metric with the following parameters:
- Baseline conversion (p0): 8.0% (0.08)
- Relative Minimum Detectable Effect (MDE): 8% (This implies p1 = 1.08 × p0 = 0.0864, resulting in an absolute lift Δ = 0.0064)
- Two-sided alpha (α): 0.05
- Statistical power: 80% (β = 0.20)
- Allocation ratio: 1:1 (equal sample sizes per arm)
Assume a z-test for the difference in proportions using a normal approximation.
Tasks
(a) Derive and compute the required per-arm sample size using the normal approximation to the difference in proportions. Clearly show the z-terms you use.
(b) Holding all other parameters constant (relative to part a), qualitatively and quantitatively describe how the required sample size changes when each of the following is modified individually:
- MDE halves.
- α is tightened to 0.01 (two-sided).
- Power is increased to 90%.
- Allocation is 75/25 (treatment/control).
- Observations are clustered at the user level with an Intraclass Correlation Coefficient (ICC) = 0.02 and an average cluster size (m) = 5. Compute the design effect and apply it.
- You adopt group-sequential monitoring with two equally spaced looks using O’Brien-Fleming boundaries.
(c) If the primary metric is an overdispersed count (Negative Binomial 2, NB2), outline how you would re-estimate the required sample size.