Rhetoric, Logic, and Dialectic:Advancing Theory-based Argument Quality Assessmentin Natural Language Processing
Table 1: Number of annotators per instance and total instances annotated by Experts and the Crowd, and the number of overlapping instances by domain.
['Domain', 'Total', 'Train', 'Dev.', 'Test']
qualifications and pay?color= yellow!20todo : color=yellow!20crowd workers were all native English speakers paid $.45-$.65 per judgment (increased over time) to ensure $21 per hour. manually reviewed each crowd workers judgments to block those underperforming and used MACE and Krippendorff to assess annotators -LN For both groups, we used the same annotation guidelines and annotation interface, which we iteratively improved according to the feedback collected in each calibration round. The guidelines as well as our interface are available online. For each domain, up to 500 arguments were annotated by both experts and crowd workers.color=blue!20todo: color=blue!20Just to confirm: 1,2,3 in Experts the number of Experts that annotated those n’s. So in column 3, 625 Q&A arguments were annotated by 3 experts. Is this right? Yes! be in the Standard Splits section