Sign in to track progress, unlock prep layers, and browse your pipeline.Sign in
Design offline backtest and online experiment
Problem Description
You are tasked with designing a system to identify and control ACH transaction fraud. The current date is 2025-09-01. Your deliverables include an offline analysis plan and an online experiment plan for launching a new fraud policy or heuristic.
Deliverables
- Offline Analysis Plan and Backtest: Cover data from 2025-06-01 to 2025-08-31.
- Online Experiment Plan: For the launch of a new fraud policy or heuristic.
Requirements
- Labeling: Define fraud labels using
ach_returnswith a 5-business-day label window. You must prevent look-ahead bias and correctly handle late returns and partial reversals. - Features/Heuristics: Propose 3–5 interpretable rules for fraud detection. Examples include 24-hour ACH velocity, shared-device usage across multiple users, or transaction amount thresholds. You must specify the exact definitions and thresholds for each proposed rule.
- Metrics:
- Primary: Loss per 1,000 credits.
- Secondary: Fraud prevalence, precision/recall, and customer contact rate.
- Guardrails: ACH acceptance rate, support ticket rate, and chargeback/return rate on non-ACH methods.
- Offline Backtest: Describe your approach to sampling, cross-validation (or time-split methodology), and leakage checks. Explain how you will simulate holds/blocks without affecting historical behavior. Detail how you will estimate counterfactuals and quantify uncertainty.
- Heterogeneity: Identify 3 distinct cuts (e.g., customer tenure, country, device clusters) and explain how you will control for false discovery rates across these segments.
- Experiment Design: Specify the unit of randomization (e.g., user-level sticky assignment), inputs for power analysis, a ramp schedule for the experiment, strategies for handling interference/spillover, methods for conducting SRM (Sample Ratio Mismatch) checks, and pre-specified stop/rollback criteria.
- Monitoring: Outline daily dashboards, anomaly detection strategies, and how you will differentiate seasonal effects (e.g., end-of-month payroll surges) from actual treatment effects.
- Presentation: Outline the content for 3 slides you would present: "Problem & Baseline," "Proposed Controls & Risk," and "Expected Value & Ramp Plan." For each slide, specify the exact readouts you expect to show.
AI interview prep
Core dilemma
Balancing the immediate financial imperative of fraud reduction with the long-term strategic goal of maintaining a positive customer experience and high transaction acceptance rates, all while navigating the complexities of time-dependent data, causal inference, and operationalizing real-time decisions.
Mid-level signals3
Senior signals3
Red flags3
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design offline backtest and online experiment
Problem Description
You are tasked with designing a system to identify and control ACH transaction fraud. The current date is 2025-09-01. Your deliverables include an offline analysis plan and an online experiment plan for launching a new fraud policy or heuristic.
Deliverables
- Offline Analysis Plan and Backtest: Cover data from 2025-06-01 to 2025-08-31.
- Online Experiment Plan: For the launch of a new fraud policy or heuristic.
Requirements
- Labeling: Define fraud labels using
ach_returnswith a 5-business-day label window. You must prevent look-ahead bias and correctly handle late returns and partial reversals. - Features/Heuristics: Propose 3–5 interpretable rules for fraud detection. Examples include 24-hour ACH velocity, shared-device usage across multiple users, or transaction amount thresholds. You must specify the exact definitions and thresholds for each proposed rule.
- Metrics:
- Primary: Loss per 1,000 credits.
- Secondary: Fraud prevalence, precision/recall, and customer contact rate.
- Guardrails: ACH acceptance rate, support ticket rate, and chargeback/return rate on non-ACH methods.
- Offline Backtest: Describe your approach to sampling, cross-validation (or time-split methodology), and leakage checks. Explain how you will simulate holds/blocks without affecting historical behavior. Detail how you will estimate counterfactuals and quantify uncertainty.
- Heterogeneity: Identify 3 distinct cuts (e.g., customer tenure, country, device clusters) and explain how you will control for false discovery rates across these segments.
- Experiment Design: Specify the unit of randomization (e.g., user-level sticky assignment), inputs for power analysis, a ramp schedule for the experiment, strategies for handling interference/spillover, methods for conducting SRM (Sample Ratio Mismatch) checks, and pre-specified stop/rollback criteria.
- Monitoring: Outline daily dashboards, anomaly detection strategies, and how you will differentiate seasonal effects (e.g., end-of-month payroll surges) from actual treatment effects.
- Presentation: Outline the content for 3 slides you would present: "Problem & Baseline," "Proposed Controls & Risk," and "Expected Value & Ramp Plan." For each slide, specify the exact readouts you expect to show.