Building Low-Resource NER Models Using Non-Speaker Annotations
Table 3: Annotation quality of annotations collected from fluent speaker (FS) and non-speaker (NS) annotators against the gold data.
['Language', 'FS P', 'FS R', 'FS  [ITALIC] F1', 'NS P', 'NS R', 'NS  [ITALIC] F1']
Annotation Quality refers to the results of participant annotation compared to the existing gold annotations on the same documents. In this evaluation, no model is trained, and we simply calculate the F1 scores by treating NS annotations as predictions themselves. Unsurprisingly, the F1 scores of FS data vastly exceed scores of NS data. There is also an expected decrease in NS scores correlated with perceived language difficulty, with Indonesian having the highest quality, and Hindi having the lowest.