Build and evaluate imbalanced binary classifier
Imbalanced Binary Classification with Temporal Split, Calibration, and Operating Point Selection
You are provided with an event-level dataset for a binary classification problem exhibiting severe class imbalance (positive rate approximately 1%). The objective is to construct a reproducible modeling pipeline, evaluate its performance using appropriate metrics, and propose a principled operating point for deployment.
Data Description: Each row contains:
id: Unique identifier.event_date: The date of the event (YYYY-MM-DD).region: Geographical region (NA, EU, APAC, LATAM, MEA).f1tof50: 50 numerical features.y: The binary label (0 or 1).
Temporal Splits:
- Train:
event_date<= 2025-06-01 - Validation: 2025-06-02 to 2025-08-01
- Test: 2025-08-02 to 2025-09-01
Tasks: (a) Develop a reproducible training pipeline that includes:
- Implementing the specified temporal split.
- Applying
StandardScalerto numerical features andOneHotEncoderto theregionfeature. - Addressing class imbalance within cross-validation folds (e.g., using
class_weight='balanced'or SMOTE/SMOTENC, ensuring no validation data leakage). - Training a robust baseline model (e.g., calibrated logistic regression or gradient boosting) that outputs well-calibrated probabilities, with calibration performed on the validation split.
(b) Report the following on the test split:
- ROC-AUC and PR-AUC.
- Recall achieved at a False Positive Rate (FPR) of 5%.
- The decision threshold that maximizes the F1 score, subject to a minimum recall of 90%. This threshold should be selected using the validation set, and its corresponding test performance reported.
(c) Describe the process for selecting the operating point for a production system that has a strict requirement of no more than 2 false positives per 1,000 predictions.
(d) Discuss potential temporal drift in model calibration and propose a technique for monitoring and re-calibrating the model without introducing label leakage.
AI interview prep
Core dilemma
Balancing the detection of rare positive events (high recall) against the cost of false positives in a severely imbalanced classification problem, while ensuring model reliability and robustness to temporal shifts.
3 tradeoffs and 2 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Build and evaluate imbalanced binary classifier
Imbalanced Binary Classification with Temporal Split, Calibration, and Operating Point Selection
You are provided with an event-level dataset for a binary classification problem exhibiting severe class imbalance (positive rate approximately 1%). The objective is to construct a reproducible modeling pipeline, evaluate its performance using appropriate metrics, and propose a principled operating point for deployment.
Data Description: Each row contains:
id: Unique identifier.event_date: The date of the event (YYYY-MM-DD).region: Geographical region (NA, EU, APAC, LATAM, MEA).f1tof50: 50 numerical features.y: The binary label (0 or 1).
Temporal Splits:
- Train:
event_date<= 2025-06-01 - Validation: 2025-06-02 to 2025-08-01
- Test: 2025-08-02 to 2025-09-01
Tasks: (a) Develop a reproducible training pipeline that includes:
- Implementing the specified temporal split.
- Applying
StandardScalerto numerical features andOneHotEncoderto theregionfeature. - Addressing class imbalance within cross-validation folds (e.g., using
class_weight='balanced'or SMOTE/SMOTENC, ensuring no validation data leakage). - Training a robust baseline model (e.g., calibrated logistic regression or gradient boosting) that outputs well-calibrated probabilities, with calibration performed on the validation split.
(b) Report the following on the test split:
- ROC-AUC and PR-AUC.
- Recall achieved at a False Positive Rate (FPR) of 5%.
- The decision threshold that maximizes the F1 score, subject to a minimum recall of 90%. This threshold should be selected using the validation set, and its corresponding test performance reported.
(c) Describe the process for selecting the operating point for a production system that has a strict requirement of no more than 2 false positives per 1,000 predictions.
(d) Discuss potential temporal drift in model calibration and propose a technique for monitoring and re-calibrating the model without introducing label leakage.