Sign in to track progress, unlock prep layers, and browse your pipeline.Sign in
Design an A/B test with causal inference
A/B Test Design: Checkout Nudge (Guest-Level Randomization)
Setup
Design an A/B test for a "checkout nudge" feature. The experiment parameters are as follows:
- Run Dates: 2025-08-04 to 2025-08-31 (28 days).
- Analysis Panel: Primary metric analysis should be on a matured panel (first exposures through 2025-08-24 to allow a 7-day lookback), or allow a 7-day measurement lag to 2025-09-07.
- Randomization Unit:
guest_id(sticky across sessions during the test). - Primary Metric: Completed order within 7 days of first exposure to the checkout nudge.
- Guardrails: Bounce rate; p95 page latency.
- Baseline 7-day Conversion per Guest: 5%.
- Minimum Detectable Effect (MDE): 8% relative lift over baseline.
- Test Parameters: Two-sided α = 0.05; power = 0.80.
- Average Sessions per Guest: 1.6.
- Intra-class Correlation (ICC) across sessions within a guest: 0.05.
- Constraints: The system must account for repeat visitors across devices, approximately 5% bot traffic, and some cookie resets causing cross-arm contamination.
Tasks
- Define the estimand (ITT vs TOT), justify guest vs session as the analysis unit, and define exposure with cross-device deduping and noncompliance handling.
- Compute required per-arm sample size accounting for clustering. Show the design effect and the final required
nper arm. - Specify SRM (Sample Ratio Mismatch) and integrity checks (e.g., device/geo imbalance, traffic-source mix), how to detect them, and how to remediate issues.
- If randomization fails and you only have pre/post windows (pre: 2025-07-01–2025-07-31; post: 2025-09-01–2025-09-30), propose a credible causal strategy (e.g., DiD with covariates/CUPED or PSM/IPW). State the identifying assumptions, write the ATE estimator, and describe tests for parallel trends and overlap.
- Address interference/novelty and propose a sequential monitoring plan that controls Type I error (e.Brien–Fleming boundaries) and a plan for early stopping for harm.
- Suppose the experiment ends with control conversion = 5.0% (n = 120,000) and treatment conversion = 5.6% (n = 120,000). Compute the lift, its standard error and 95% CI for a two-proportion comparison, interpret statistical and practical significance, and state whether you would ship.
AI interview prep
Core dilemma
Ensuring statistical validity and practical interpretability of A/B test results in a complex, real-world system with inherent challenges like clustering, contamination, and potential randomization failures.
Mid-level signals3
Senior signals3
Red flags3
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design an A/B test with causal inference
A/B Test Design: Checkout Nudge (Guest-Level Randomization)
Setup
Design an A/B test for a "checkout nudge" feature. The experiment parameters are as follows:
- Run Dates: 2025-08-04 to 2025-08-31 (28 days).
- Analysis Panel: Primary metric analysis should be on a matured panel (first exposures through 2025-08-24 to allow a 7-day lookback), or allow a 7-day measurement lag to 2025-09-07.
- Randomization Unit:
guest_id(sticky across sessions during the test). - Primary Metric: Completed order within 7 days of first exposure to the checkout nudge.
- Guardrails: Bounce rate; p95 page latency.
- Baseline 7-day Conversion per Guest: 5%.
- Minimum Detectable Effect (MDE): 8% relative lift over baseline.
- Test Parameters: Two-sided α = 0.05; power = 0.80.
- Average Sessions per Guest: 1.6.
- Intra-class Correlation (ICC) across sessions within a guest: 0.05.
- Constraints: The system must account for repeat visitors across devices, approximately 5% bot traffic, and some cookie resets causing cross-arm contamination.
Tasks
- Define the estimand (ITT vs TOT), justify guest vs session as the analysis unit, and define exposure with cross-device deduping and noncompliance handling.
- Compute required per-arm sample size accounting for clustering. Show the design effect and the final required
nper arm. - Specify SRM (Sample Ratio Mismatch) and integrity checks (e.g., device/geo imbalance, traffic-source mix), how to detect them, and how to remediate issues.
- If randomization fails and you only have pre/post windows (pre: 2025-07-01–2025-07-31; post: 2025-09-01–2025-09-30), propose a credible causal strategy (e.g., DiD with covariates/CUPED or PSM/IPW). State the identifying assumptions, write the ATE estimator, and describe tests for parallel trends and overlap.
- Address interference/novelty and propose a sequential monitoring plan that controls Type I error (e.Brien–Fleming boundaries) and a plan for early stopping for harm.
- Suppose the experiment ends with control conversion = 5.0% (n = 120,000) and treatment conversion = 5.6% (n = 120,000). Compute the lift, its standard error and 95% CI for a two-proportion comparison, interpret statistical and practical significance, and state whether you would ship.