Compute power and cost-aware thresholds
Evaluating an ACH Fraud Block Rule
You are tasked with evaluating a new Automated Clearing House (ACH) velocity and shared-device block rule designed to mitigate fraud. Your analysis should cover the financial impact, experimental design, and statistical inference.
Assumptions
- Volume: 1,000,000 ACH credits per month.
- Baseline Fraud Prevalence: 0.15% of credits are identified as fraudulent within 5 business days.
- Mean Loss per Fraudulent Credit: $900, with a standard deviation (SD) of $600.
- Proposed Rule Efficacy:
- Recalls 70% of fraudulent credits.
- Incurs a False Positive Rate (FPR) of 0.05% on legitimate credits.
- Cost per False Positive: $25 (includes operational costs and potential customer churn).
Tasks
A) Expected Value Calculation
Compute the following, showing all formulas and units:
- Monthly gross fraud loss without the proposed rule.
- Monthly gross fraud loss with the proposed rule.
- Net monthly savings after accounting for false-positive costs.
B) A/B Test Design and Sample Size
Design a two-arm online A/B test to evaluate the new rule.
- Primary Metric: "Loss per 1,000 credits".
- Target Effect Size: Detect a 15% relative reduction in the primary metric.
- Statistical Parameters:
- Significance level (α): 0.05 (two-sided).
- Power: 80%.
- Deliverables:
- State all distributional assumptions (e.g., approximate as a Poisson rate or two-part model).
- Derive the required per-arm sample size.
- Justify your choice of distributional assumptions and methodology.
C) Interval Estimation
Given an observation from the control arm:
- Observation: 240,000 credits processed, with a 0.12% fraud prevalence.
- Deliverables:
- Compute a 95% Wilson interval for the true fraud prevalence.
- Interpret this interval for decision-making purposes.
D) Sequential Monitoring
Assume the A/B test results from Task B are reviewed daily for a period of 14 days.
- Deliverables:
- Propose a valid sequential testing plan (e.g., using alpha-spending functions or group-sequential boundaries).
- Define appropriate stopping rules that effectively control the Type I error rate.
- Explain how early stopping might interact with the chosen primary metric ("Loss per 1,000 credits").
AI interview prep
Core dilemma
Balancing the statistical rigor and validity of experimental design with the practical constraints of real-world data, cost, and time-to-decision, especially when dealing with rare events and high-variance outcomes like fraud loss.
3 tradeoffs and 2 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Compute power and cost-aware thresholds
Evaluating an ACH Fraud Block Rule
You are tasked with evaluating a new Automated Clearing House (ACH) velocity and shared-device block rule designed to mitigate fraud. Your analysis should cover the financial impact, experimental design, and statistical inference.
Assumptions
- Volume: 1,000,000 ACH credits per month.
- Baseline Fraud Prevalence: 0.15% of credits are identified as fraudulent within 5 business days.
- Mean Loss per Fraudulent Credit: $900, with a standard deviation (SD) of $600.
- Proposed Rule Efficacy:
- Recalls 70% of fraudulent credits.
- Incurs a False Positive Rate (FPR) of 0.05% on legitimate credits.
- Cost per False Positive: $25 (includes operational costs and potential customer churn).
Tasks
A) Expected Value Calculation
Compute the following, showing all formulas and units:
- Monthly gross fraud loss without the proposed rule.
- Monthly gross fraud loss with the proposed rule.
- Net monthly savings after accounting for false-positive costs.
B) A/B Test Design and Sample Size
Design a two-arm online A/B test to evaluate the new rule.
- Primary Metric: "Loss per 1,000 credits".
- Target Effect Size: Detect a 15% relative reduction in the primary metric.
- Statistical Parameters:
- Significance level (α): 0.05 (two-sided).
- Power: 80%.
- Deliverables:
- State all distributional assumptions (e.g., approximate as a Poisson rate or two-part model).
- Derive the required per-arm sample size.
- Justify your choice of distributional assumptions and methodology.
C) Interval Estimation
Given an observation from the control arm:
- Observation: 240,000 credits processed, with a 0.12% fraud prevalence.
- Deliverables:
- Compute a 95% Wilson interval for the true fraud prevalence.
- Interpret this interval for decision-making purposes.
D) Sequential Monitoring
Assume the A/B test results from Task B are reviewed daily for a period of 14 days.
- Deliverables:
- Propose a valid sequential testing plan (e.g., using alpha-spending functions or group-sequential boundaries).
- Define appropriate stopping rules that effectively control the Type I error rate.
- Explain how early stopping might interact with the chosen primary metric ("Loss per 1,000 credits").