Compute robust inference under skew and outliers
A/B Test with Skew, Outliers, Heteroskedasticity, Missingness, and Multiplicity
You are tasked with comparing two independent product variants (A and B) based on a continuous Key Performance Indicator (KPI), specifically session revenue in USD. The KPI distribution is known to be right-skewed and contains approximately 5% extreme outliers.
Historical data provides the following estimates:
- Variant A: mean μ_A ≈ 2.30, standard deviation σ_A ≈ 1.10
- Variant B: mean μ_B ≈ 2.35, standard deviation σ_B ≈ 1.40
Planned sample sizes are:
- Variant A: n_A = 8,000
- Variant B: n_B = 7,500
An expected 10% of data in Variant B is missing completely at random (MCAR).
Your tasks are:
- Test Choice: Select an appropriate statistical test (Welch t-test on means, Mann–Whitney test on distributions, or a trimmed-mean test). Justify your choice based on robustness to skew and outliers, and the specific parameter you aim to estimate.
- Confidence Interval for Mean Difference: Compute a 95% confidence interval (CI) for the difference in means (μ_B − μ_A) using Welch's t-test, including the Satterthwaite degrees of freedom. Additionally, describe how to compute a nonparametric bootstrap percentile 95% CI using 10,000 resamples, and discuss the pros and cons of each method.
- Power Approximation: Approximate the statistical power to detect a true mean lift of +0.05 at a significance level α = 0.05. Assume heteroskedasticity (using Welch's variance) and account for the 10% MCAR data in Variant B.
- Robust Estimation: Propose an outlier-robust estimator (e.g., a 20% trimmed mean or a Huber M-estimator) and compute a 98% CI for the trimmed-mean difference. Explain the process for selecting the trimming proportion.
- Multiple Testing: If you were tracking 20 related KPIs, describe and compute the Benjamini–Hochberg False Discovery Rate (FDR) control at q = 0.10, given sorted p-values p_(1) ≤ … ≤ p_(20). Contrast this with the Bonferroni correction in terms of power and Type I error.
- Variance-Stabilizing Transform: If normality assumptions are questionable, demonstrate how to use a Y' = log1p(Y) transform. Show how to back-transform the CI to obtain an interpretable multiplicative effect, noting any necessary bias corrections when exponentiating.
AI interview prep
Core dilemma
Balancing statistical rigor and interpretability in the presence of complex data imperfections (skew, outliers, missingness, multiple comparisons) to make a reliable business decision from A/B test results.
4 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Compute robust inference under skew and outliers
A/B Test with Skew, Outliers, Heteroskedasticity, Missingness, and Multiplicity
You are tasked with comparing two independent product variants (A and B) based on a continuous Key Performance Indicator (KPI), specifically session revenue in USD. The KPI distribution is known to be right-skewed and contains approximately 5% extreme outliers.
Historical data provides the following estimates:
- Variant A: mean μ_A ≈ 2.30, standard deviation σ_A ≈ 1.10
- Variant B: mean μ_B ≈ 2.35, standard deviation σ_B ≈ 1.40
Planned sample sizes are:
- Variant A: n_A = 8,000
- Variant B: n_B = 7,500
An expected 10% of data in Variant B is missing completely at random (MCAR).
Your tasks are:
- Test Choice: Select an appropriate statistical test (Welch t-test on means, Mann–Whitney test on distributions, or a trimmed-mean test). Justify your choice based on robustness to skew and outliers, and the specific parameter you aim to estimate.
- Confidence Interval for Mean Difference: Compute a 95% confidence interval (CI) for the difference in means (μ_B − μ_A) using Welch's t-test, including the Satterthwaite degrees of freedom. Additionally, describe how to compute a nonparametric bootstrap percentile 95% CI using 10,000 resamples, and discuss the pros and cons of each method.
- Power Approximation: Approximate the statistical power to detect a true mean lift of +0.05 at a significance level α = 0.05. Assume heteroskedasticity (using Welch's variance) and account for the 10% MCAR data in Variant B.
- Robust Estimation: Propose an outlier-robust estimator (e.g., a 20% trimmed mean or a Huber M-estimator) and compute a 98% CI for the trimmed-mean difference. Explain the process for selecting the trimming proportion.
- Multiple Testing: If you were tracking 20 related KPIs, describe and compute the Benjamini–Hochberg False Discovery Rate (FDR) control at q = 0.10, given sorted p-values p_(1) ≤ … ≤ p_(20). Contrast this with the Bonferroni correction in terms of power and Type I error.
- Variance-Stabilizing Transform: If normality assumptions are questionable, demonstrate how to use a Y' = log1p(Y) transform. Show how to back-transform the CI to obtain an interpretable multiplicative effect, noting any necessary bias corrections when exponentiating.