Design and sample for credit default prediction
A bank aims to develop a machine learning model to predict 90-day credit card default at an account-month level. This model will facilitate proactive outreach to at-risk customers. The true class prevalence in production is approximately 2% defaults.
Your task is to design the end-to-end approach for this model, with a particular focus on the sampling strategy.
a) Problem Framing
Define the label precisely, including the observation window, prediction date, and prediction horizon. List the features that would be available at the time of scoring. Propose a temporal data split strategy that effectively avoids leakage (e.g., training on data up to a specific cutoff date and validating on future months). Identify three concrete leakage risks unique to credit card data and explain how you would detect them.
b) Metrics
Select appropriate evaluation metrics and operating points for imbalanced data (e.g., PR-AUC, recall at 5% FPR, expected utility). Justify your choices by explaining how they align with the business goals, considering the 2% default prevalence.
c) Sampling Strategy and Calibration
Assume you downsample negative instances to accelerate training, targeting a 1:3 positive-to-negative ratio in the training set. Let π represent the true prevalence (0.02 in production) and π_s be the training prevalence (0.25 after downsampling). If the model outputs a sampled probability p_s = P(default | x, sampled) = 0.60 for a given account, derive and compute the calibrated population probability p_pop = P(default | x) using prior-probability correction under prior shift, following these formulas:
odds_s = p_s / (1 - p_s)odds_pop = odds_s × [(π / (1 - π)) / (π_s / (1 - π_s))]p_pop = odds_pop / (1 + odds_pop)
Provide the numeric value for p_pop. Additionally, explain the circumstances under which class weights alone, probability recalibration alone, or both are necessary.
d) Threshold via Cost-Benefit Analysis
An outreach intervention costs $2 per account. If an account would default and you contact them, there is a 30% chance that the intervention successfully averts an average loss of $150. Choose an action rule and compute the breakeven probability threshold p* for contacting an account. Based on this threshold, decide whether to contact the account from part (c) using its calculated p_pop.
e) Validation
Propose a time-based cross-validation scheme. Describe a backtesting approach that demonstrates model stability under covariate and prior shift across different regions and macroeconomic regimes. Detail how you would monitor calibration and drift post-deployment and how you would re-tune the sampling strategy if the true default rate changes from 2% to 1%.
f) Fairness and Operations
Name two fairness checks (e.g., equal opportunity across age groups, where legally permitted) that you would implement. Propose one operational guardrail to prevent over-throttling credit limits due to model uncertainty. Explain how sampling choices, if not properly corrected, can introduce bias into these fairness checks.
AI interview prep
Core dilemma
Designing a robust, fair, and economically optimal machine learning system for a rare event (credit default) under significant data imbalance, temporal dynamics, and real-world operational constraints.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design and sample for credit default prediction
A bank aims to develop a machine learning model to predict 90-day credit card default at an account-month level. This model will facilitate proactive outreach to at-risk customers. The true class prevalence in production is approximately 2% defaults.
Your task is to design the end-to-end approach for this model, with a particular focus on the sampling strategy.
a) Problem Framing
Define the label precisely, including the observation window, prediction date, and prediction horizon. List the features that would be available at the time of scoring. Propose a temporal data split strategy that effectively avoids leakage (e.g., training on data up to a specific cutoff date and validating on future months). Identify three concrete leakage risks unique to credit card data and explain how you would detect them.
b) Metrics
Select appropriate evaluation metrics and operating points for imbalanced data (e.g., PR-AUC, recall at 5% FPR, expected utility). Justify your choices by explaining how they align with the business goals, considering the 2% default prevalence.
c) Sampling Strategy and Calibration
Assume you downsample negative instances to accelerate training, targeting a 1:3 positive-to-negative ratio in the training set. Let π represent the true prevalence (0.02 in production) and π_s be the training prevalence (0.25 after downsampling). If the model outputs a sampled probability p_s = P(default | x, sampled) = 0.60 for a given account, derive and compute the calibrated population probability p_pop = P(default | x) using prior-probability correction under prior shift, following these formulas:
odds_s = p_s / (1 - p_s)odds_pop = odds_s × [(π / (1 - π)) / (π_s / (1 - π_s))]p_pop = odds_pop / (1 + odds_pop)
Provide the numeric value for p_pop. Additionally, explain the circumstances under which class weights alone, probability recalibration alone, or both are necessary.
d) Threshold via Cost-Benefit Analysis
An outreach intervention costs $2 per account. If an account would default and you contact them, there is a 30% chance that the intervention successfully averts an average loss of $150. Choose an action rule and compute the breakeven probability threshold p* for contacting an account. Based on this threshold, decide whether to contact the account from part (c) using its calculated p_pop.
e) Validation
Propose a time-based cross-validation scheme. Describe a backtesting approach that demonstrates model stability under covariate and prior shift across different regions and macroeconomic regimes. Detail how you would monitor calibration and drift post-deployment and how you would re-tune the sampling strategy if the true default rate changes from 2% to 1%.
f) Fairness and Operations
Name two fairness checks (e.g., equal opportunity across age groups, where legally permitted) that you would implement. Propose one operational guardrail to prevent over-throttling credit limits due to model uncertainty. Explain how sampling choices, if not properly corrected, can introduce bias into these fairness checks.