Racial Bias in Hate Speech and Abusive Language Detection Datasets
Table 2: Experiment 1
['Dataset', 'Class', 'ˆ [ITALIC] piblack', 'ˆ [ITALIC] piwhite', '[ITALIC] t', '[ITALIC] p', 'ˆ [ITALIC] piblackˆ [ITALIC] piwhite']
The results of Experiment 1 are shown in Table 2. [CONTINUE] We observe substantial racial disparities in the performance of all classifiers. In all but one of the comparisons, there are statistically significant (p < 0.001) differences and in all but one of these we see that tweets in the black-aligned corpus are assigned negative labels more frequently than those by whites. The only case where blackaligned tweets are classified into a negative class less frequently than white-aligned tweets is the racism class in the Waseem and Hovy (2016) classifier. Note, however, the extremely low rate at which tweets are predicted to belong to this class for both groups. On the other hand, this classifier is 1.7 times more likely to classify tweets in the black-aligned corpus as sexist. For Waseem (2016) we see that there is no significant difference in the estimated rates at which tweets are clas [CONTINUE] sified as racist across groups, although the rates remain low. Tweets in the black-aligned corpus are classified as containing sexism almost twice as frequently and 1.1 times as frequently classified as containing racism and sexism compared to those in the white-aligned corpus. Moving onto Davidson et al. (2017), we find large disparities, with around 5% of tweets in the black-aligned corpus classified as hate speech compared to 2% of those in the white-aligned set. Similarly, 17% of black-aligned tweets are predicted to contain offensive language compared to 6.5% of whitealigned tweets. The classifier trained on the Golbeck et al. (2017) dataset predicts black-aligned tweets to be harassment 1.4 times as frequently as white-aligned tweets. The Founta et al. (2018) classifier labels around 11% of tweets in the blackaligned corpus as hate speech and almost 18% as abusive, compared to 6% and 8% of white-aligned tweets respectively. It also classifies black-aligned tweets as spam 1.8 times as frequently. The results of Experiment 2 are consistent with the previous results, although there are some notable differences. In most cases the racial disparities persist, although they are generally smaller in magnitude and in some cases the direction even changes.