Compute sample sizes and error control
A/B Experiment Design: Sample Size and Error Control
This problem explores the design of A/B experiments, focusing on sample size calculations and error control strategies under various practical constraints, using the context of a "Biker experiment." You should show formulas and numeric answers where possible.
Assumptions
- Two-armed test (control vs. Biker exposure).
- Two-sided alpha unless explicitly stated otherwise.
Questions
-
Primary Mean Metric:
- Baseline mean delivery time: 42 minutes, Standard Deviation (SD): 15 minutes.
- Target relative improvement: -5%.
- Alpha (two-sided): 0.05.
- Power: 0.80.
- Compute the per-arm sample size required for a two-sample t-test on means.
-
Guardrail Proportion Metric:
- Baseline cancellation rate: 6%.
- Requirement: Non-inferiority with a margin of +0.5 percentage points (i.e., Biker cancel rate ≤ 6.5%).
- Alpha (one-sided): 0.05.
- Power: 0.80.
- Compute the per-arm sample size required for a non-inferiority test on proportions.
-
Multiple Metrics Error Control:
- You are tracking 1 primary metric, 2 guardrail metrics (cancellations, ETA accuracy), and 1 secondary metric (orders per courier-hour).
- Propose and justify an error-control approach (e.g., Bonferroni, Holm/Hochberg, gatekeeping/HMP).
- State the effective alpha for each family of metrics and explain how you would report adjusted confidence intervals (CIs).
-
Cluster Randomization:
- Randomization is at the zone-day level.
- Average orders per cluster (m): 300.
- Intraclass Correlation Coefficient (ICC) for the primary metric: 0.03.
- Compute the Design Effect (DE) and the adjusted per-arm sample size.
- Determine how many zone-days per arm are needed to achieve this adjusted sample size.
-
Sequential Monitoring:
- You plan 4 equally spaced looks with O’Brien–Fleming spending.
- Qualitatively explain how early critical values differ from the final look and how this affects runtime/Minimum Detectable Effect (MDE).
- Provide the final-look alpha spending approximation.
- Describe how you would implement boundary checks in practice.
AI interview prep
Core dilemma
Balancing the statistical power to detect meaningful effects with the practical constraints of experiment duration, resource cost, and the risk of false positives (Type I errors) or false negatives (Type II errors) across multiple, potentially correlated, metrics.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Compute sample sizes and error control
A/B Experiment Design: Sample Size and Error Control
This problem explores the design of A/B experiments, focusing on sample size calculations and error control strategies under various practical constraints, using the context of a "Biker experiment." You should show formulas and numeric answers where possible.
Assumptions
- Two-armed test (control vs. Biker exposure).
- Two-sided alpha unless explicitly stated otherwise.
Questions
-
Primary Mean Metric:
- Baseline mean delivery time: 42 minutes, Standard Deviation (SD): 15 minutes.
- Target relative improvement: -5%.
- Alpha (two-sided): 0.05.
- Power: 0.80.
- Compute the per-arm sample size required for a two-sample t-test on means.
-
Guardrail Proportion Metric:
- Baseline cancellation rate: 6%.
- Requirement: Non-inferiority with a margin of +0.5 percentage points (i.e., Biker cancel rate ≤ 6.5%).
- Alpha (one-sided): 0.05.
- Power: 0.80.
- Compute the per-arm sample size required for a non-inferiority test on proportions.
-
Multiple Metrics Error Control:
- You are tracking 1 primary metric, 2 guardrail metrics (cancellations, ETA accuracy), and 1 secondary metric (orders per courier-hour).
- Propose and justify an error-control approach (e.g., Bonferroni, Holm/Hochberg, gatekeeping/HMP).
- State the effective alpha for each family of metrics and explain how you would report adjusted confidence intervals (CIs).
-
Cluster Randomization:
- Randomization is at the zone-day level.
- Average orders per cluster (m): 300.
- Intraclass Correlation Coefficient (ICC) for the primary metric: 0.03.
- Compute the Design Effect (DE) and the adjusted per-arm sample size.
- Determine how many zone-days per arm are needed to achieve this adjusted sample size.
-
Sequential Monitoring:
- You plan 4 equally spaced looks with O’Brien–Fleming spending.
- Qualitatively explain how early critical values differ from the final look and how this affects runtime/Minimum Detectable Effect (MDE).
- Provide the final-look alpha spending approximation.
- Describe how you would implement boundary checks in practice.