Rhetoric, Logic, and Dialectic:Advancing Theory-based Argument Quality Assessmentin Natural Language Processing
Table 3: Agreement between expert annotations from wachsmuth-etal-2017-computational and crowd-sourced annotations from two sources: AQCorpus (Ours) and wachsmuth-etal-2017-argumentation (TvsP) on 200 randomly sampled instances.
['Domain', 'IAA Expert/Crowd ( [ITALIC] α) Cog', 'IAA Expert/Crowd ( [ITALIC] α) Effec', 'IAA Expert/Crowd ( [ITALIC] α) Reas', 'IAA Expert/Crowd ( [ITALIC] α) Over']
Inter-annotator Agreements. In order to assess the quality of our crowd-sourced annotations and to test our simplified annotation guidelines, we employ the Dagstuhl-ArgQuality-Corpus-V2 (DS) We take “gold“ ratings from the original, author-produced annotations presented in \newcitewachsmuth-etal-2017-computational. DS was presented in combination with the taxonomy of theory-based AQ described above and consists of 320 web debate arguments annotated with all 15 argument quality aspects. We randomly sample 200 arguments and crowd-source annotations on Appen.color= blue!20todo : color= blue!20how many workers were used per argument? is it the same as in the prior section? For each instance and AQ dimension, we obtain group votes by either averaging the workers’ votes (Mean) or applying MACEcolor=green!20todo: color =green!20why do we use MACE, and do we need to? We measure inter-annotator agreement (IAA) between the obtained group vote and the DS expert vote. Generally, the agreement scores of our crowd ratings are comparably high and often surpass the agreement scores reported by TvsP. We therefore conclude that our guidelines and user interface support the task, and confirm the suitability of our crowd annotators. For debate forums, Krippendorff’s α is up to .21 , while for the Q&A forums, the agreement is higher – up to .53. These results suggest that the difficulty of the task is highly dependent on the domain.