Crowdsourcing Argumentation Structures in Chinese Hotel Reviews
TABLE II: The average αU for students’ annotations on gold standard texts, before and after the removal of less-devoted students’ annotations.
['[EMPTY]', '[BOLD] gold 1', '[BOLD] gold 2', '[BOLD] gold 3', '[BOLD] gold 4', '[BOLD] gold 5']
We find that removing the less-devoted students’ annotations can indeed improve the annotation quality as a whole. For example, after removal, the percentage of reviews whose αU≥0.5 increases from 21% to 24%. However, even after the removal, there still exist considerable controversial annotations, due to the controversial nature of our guideline and the argument annotation task itself. Next, we will identify the controversial texts, remove their annotations and so as to obtain high-quality corpora.