Garbage In, Garbage Out?
Table 5. Original human annotation source
['[EMPTY]', 'Count', 'Proportion']
The plurality of papers involved the paper’s authors doing the annotation work themselves. The next highest category was “no information,” which was found in almost a quarter of the papers using original human annotation. Experts / professionals was far higher than we expected, although we took any claim of expertise for granted. Crowdworkers constituted a far smaller proportion than we expected, with Amazon Mechanical Turk and other platforms collectively comprising about 15% of papers. Almost all of the other crowdworking platforms specified were CrowdFlower/FigureEight, with one paper using oDesk.