CrossWeigh: Training Named Entity Tagger from Imperfect Annotations
Table 9: Quality of noise estimation. The number of true mistakes, based on our manual correction, is 186. The potential mistakes are counted based on average of 3 runs.
['Algorithm', 'Potential Mistakes', 'Actual Mistakes', 'Precision', 'Recall', 'F1']
The potential mistakes are the total number of mistakes identified by CrossWeigh, and actual mistakes is the true positives among all identifications. From the results, we can see that when the base model is Flair, CrossWeigh is able to spot more than 75% of label mistakes, while maintaining a precision about 25%. It is worth noting that 25% is a reasonably high precision, given that the label mistake ratio is only 5.38%. The 75% recall indicates that CrossWeigh is able to identify most of the label mistakes, which are extremely valuable to improve the annotation quality.