On Analyzing Annotation Consistency in Online Abusive Behavior Datasets
Table 5: Annotation Inconsistency matrix for FOUNTA
['[EMPTY]', '[EMPTY]', '[BOLD] Contentious Tweet Label Abusive', '[BOLD] Contentious Tweet Label Hate', '[BOLD] Contentious Tweet Label Spam', '[BOLD] Contentious Tweet Label Normal']
As discussed earlier in the section, there could be multiple reasons for the misclassification. For instance, the hate speech detection problem may be hard as the tweets within the same label have high variance, or there might be insufficient training data. In this paper, we are interested to understand how much of the misclassification can be attributed to annotation inconsistency. Comparing the annotation inconsistency matrix of FOUNTA against the other two datasets, we noted that there could be significantly more annotation inconsistencies in the FOUNTA dataset. For instance, we observed that 758 contentious abusive tweets have their most similar tweets labeled as normal, and a significant number of contentious hate tweets have their most similar tweets labeled as abusive or normal. We further verify the annotation inconsistencies in FOUNTA dataset by retrieving some samples of the FOUNTA tweets. Surprisingly, we notice that the most similar tweets retrieved for contentious tweets C1 and C2 are retweets, and the retweets are annotated with different class labels. This exposes an issue in FOUNTA’s annotation strategy. We postulated that the identical tweets (i.e., the retweets) are annotated by different human annotators, resulting in the inconsistencies. We further investigated and found that more than 10% of the tweets are retweets, and most of which have inconsistencies in their annotation.