Garbage In, Garbage Out?
Table 6. Number of annotators specified
['[EMPTY]', 'Count', 'Proportion']
Based on our experiences, we typically noticed that papers discussing the number of annotators often fell into two categories: 1) a small closed team (more often 2-3, sometimes 4-6) that were either the papers’ authors or recruited directly by the authors, who tended to perform the same amount of work for the duration of the project; or 2) a medium to large (25-500) open set of annotators, typically but not necessarily recruited through a crowdworking platform, who each performed highly variable amounts of work.