Garbage In, Garbage Out?
Table 11. Reported inter-annotator agreement
['[EMPTY]', 'Count', 'Proportion']
This includes the one paper that reported inter-annotator agreement metrics, but did not specify whether overlap was for all items or some items. Only three papers explicitly stated that there was no overlap among annotators, and so it is quite likely that the papers that did not specify such information did not engage in such a practice. For the 37 papers that did involve some kind of multiple annotator overlap, the overwhelming majority of this subsample (84%) involved multiple annotation of all items, rather than only some items.