On Analyzing Annotation Consistency in Online Abusive Behavior Datasets
Table 4: Annotation Inconsistency matrix for DT
['[EMPTY]', '[EMPTY]', '[BOLD] Contentious Tweet Label Offensive', '[BOLD] Contentious Tweet Label Hate', '[BOLD] Contentious Tweet Label Neither']
As discussed earlier in the section, there could be multiple reasons for the misclassification. For instance, the hate speech detection problem may be hard as the tweets within the same label have high variance, or there might be insufficient training data. In this paper, we are interested to understand how much of the misclassification can be attributed to annotation inconsistency. This suggests that there could be inconsistencies in the annotation of sexism tweets as two similar tweets may have different labels, one labeled as sexism while another as normal. Similar observations are made in other class labels, although the inconsistency in sexism tweet annotation is observed to be the highest. A majority of the contentious hate tweets have their most similar tweets labeled as offensive. Nevertheless, such challenges in annotation also highlight the difficulty in the hate speech detection task.