Explain power drivers and resolve unexpected A/B results
A/B Testing: Power, Sample Size, Allocation, and Diagnostics
You are tasked with analyzing a two-proportion (binary conversion) A/B test. Assume independent users, no clustering or spillover effects, and equal exposure eligibility per day unless otherwise specified. Answer all parts concisely and show calculations where requested.
(a) Define Power and Rank the Levers
Define statistical power for a two-proportion A/B test. List the primary levers that increase power, ranking them by their typical practical impact (from largest to smallest). Briefly explain the trade-offs associated with each. Include the following factors:
- Effect size (MDE)
- Variance/metric volatility
- Sample size
- Allocation ratio
- Alpha
- Variance-reduction techniques (e.g., CUPED)
- Bucketing/stratification
- Test duration/seasonality
(b) Baseline Scenario: Sample Size and Duration
Given the following parameters for an A/B test:
- Baseline conversion (p0) = 5.00%
- Target relative lift = +7% (resulting in p1 = 5.35%, or a delta of 0.35 percentage points)
- Two-sided alpha = 0.05
- Desired power = 0.80
- Equal allocation (50/50 split)
- 80,000 eligible users per day
- 10% post-randomization attrition (meaning only 90% of users produce analyzable outcomes)
Compute the following:
- Required sample size per variant (in terms of analyzable users).
- Minimum test duration (in days).
Ensure to show all formulas used and the numeric results.
(c) With CUPED (R² = 0.30)
Recompute the requirements from part (b), assuming that applying CUPED (Controlled-experiment Using Pre-experiment Data) achieves a 30% relative variance reduction (R² = 0.30). What are the new required sample size per variant and the minimum test duration?
(d) Unequal Allocation (90/10)
Explain how switching to a 90/10 allocation (90% control, 10% treatment) affects statistical power, assuming a fixed total traffic. Provide an intuitive explanation and, if possible, a quantitative comparison to an equal split.
(e) Negative Significant Result: Diagnostic Plan
Imagine your A/B test, run for the duration calculated in part (b), returns a statistically significant -2% lift (meaning the treatment group performed worse), which contradicts your prior expectation of a +7% lift. Outline a step-by-step diagnostic plan you would execute before drawing any conclusions or making decisions. Your plan should address:
- SRM (Sample Ratio Mismatch) checks and their importance.
- Instrumentation and metric definition audits.
- Potential imbalances related to bots, geographical regions, or device types.
- Novelty or learning effects.
- Outlier clipping strategies.
- Simpson’s paradox, specifically how it might manifest via key segments.
- Guardrail metrics.
- Risks associated with peeking or early stopping.
(f) Decision Tree After Diagnostics
Based on the outcomes of your diagnostic plan from part (e), propose an evidence-based decision tree for the next steps. Clearly specify the conditions under which you would:
- (i) Ship the change.
- (ii) Iterate with a follow-up test (provide one concrete design change to test).
- (iii) Rerun the experiment (state the precise condition that would justify a rerun).
AI interview prep
Core dilemma
Balancing statistical rigor and validity (e.g., power, Type I/II errors) with business constraints (e.g., speed of experimentation, user experience, resource allocation, cost of data collection).
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Explain power drivers and resolve unexpected A/B results
A/B Testing: Power, Sample Size, Allocation, and Diagnostics
You are tasked with analyzing a two-proportion (binary conversion) A/B test. Assume independent users, no clustering or spillover effects, and equal exposure eligibility per day unless otherwise specified. Answer all parts concisely and show calculations where requested.
(a) Define Power and Rank the Levers
Define statistical power for a two-proportion A/B test. List the primary levers that increase power, ranking them by their typical practical impact (from largest to smallest). Briefly explain the trade-offs associated with each. Include the following factors:
- Effect size (MDE)
- Variance/metric volatility
- Sample size
- Allocation ratio
- Alpha
- Variance-reduction techniques (e.g., CUPED)
- Bucketing/stratification
- Test duration/seasonality
(b) Baseline Scenario: Sample Size and Duration
Given the following parameters for an A/B test:
- Baseline conversion (p0) = 5.00%
- Target relative lift = +7% (resulting in p1 = 5.35%, or a delta of 0.35 percentage points)
- Two-sided alpha = 0.05
- Desired power = 0.80
- Equal allocation (50/50 split)
- 80,000 eligible users per day
- 10% post-randomization attrition (meaning only 90% of users produce analyzable outcomes)
Compute the following:
- Required sample size per variant (in terms of analyzable users).
- Minimum test duration (in days).
Ensure to show all formulas used and the numeric results.
(c) With CUPED (R² = 0.30)
Recompute the requirements from part (b), assuming that applying CUPED (Controlled-experiment Using Pre-experiment Data) achieves a 30% relative variance reduction (R² = 0.30). What are the new required sample size per variant and the minimum test duration?
(d) Unequal Allocation (90/10)
Explain how switching to a 90/10 allocation (90% control, 10% treatment) affects statistical power, assuming a fixed total traffic. Provide an intuitive explanation and, if possible, a quantitative comparison to an equal split.
(e) Negative Significant Result: Diagnostic Plan
Imagine your A/B test, run for the duration calculated in part (b), returns a statistically significant -2% lift (meaning the treatment group performed worse), which contradicts your prior expectation of a +7% lift. Outline a step-by-step diagnostic plan you would execute before drawing any conclusions or making decisions. Your plan should address:
- SRM (Sample Ratio Mismatch) checks and their importance.
- Instrumentation and metric definition audits.
- Potential imbalances related to bots, geographical regions, or device types.
- Novelty or learning effects.
- Outlier clipping strategies.
- Simpson’s paradox, specifically how it might manifest via key segments.
- Guardrail metrics.
- Risks associated with peeking or early stopping.
(f) Decision Tree After Diagnostics
Based on the outcomes of your diagnostic plan from part (e), propose an evidence-based decision tree for the next steps. Clearly specify the conditions under which you would:
- (i) Ship the change.
- (ii) Iterate with a follow-up test (provide one concrete design change to test).
- (iii) Rerun the experiment (state the precise condition that would justify a rerun).