A Quality Type-aware Annotated Corpus and Lexicon for Harassment Research
Table 3. Agreement rate.
['[BOLD] Content Type', '[BOLD] Agreement Rate']
Although the annotators employed three labels, i.e., “yes”, “no”, and “other”, the eventual corpus excluded all of the tweets that did not have a consensus label of “yes” and “no”. In other words, the corpus only contains tweets that receive at least two “yes” or two “no” labels. The appearance-related context shows the highest agreement rate whereas the political and sexual contexts have the lowest indicating that these contents are more challenging to judge (ambiguity is higher).