Feature leakage in ML: Detect, Prevent, and Fix It

This post details feature leakage in ML, a critical failure mode where information not available at prediction time contaminates training data, leading to inflated validation metrics and poor production performance. It categorizes leakage into target leakage, train-test contamination, and temporal leakage. The post provides practical strategies for detecting leakage during EDA, such as suspicious feature-target correlations, near-perfect naive baselines, dominant feature importance, differential null rates, and timestamp mismatches. It also outlines pipeline-level fixes like using scikit-learn Pipelines for preprocessing, TimeSeriesSplit for temporal data, and GroupKFold for group-based validation. A new vector of leakage introduced by AI-generated code is also discussed, emphasizing the need for careful auditing of AI-assisted ML workflows.