Challenges for Toxic Comment Classification:An In-Depth Error Analysis
Table 2: Class distribution of Twitter dataset. The majority class of the dataset consists of offensive Tweets.
['Class', '# of occurrences']
Additionally we investigate a dataset introduced by Davidson et al. It contains 24,783 Tweets fetched using the Twitter API and annotated by CrowdFlower workers with the labels ‘hate speech’, ‘offensive but not hate speech’ and ‘neither offensive nor hate speech’. We observe a strong bias towards the ‘offensive’ class making up 77.4% of the comments caused by sampling tweets by seed keywords from Hatebase.org. We choose this dataset to show that our method is also applicable to multi-class problems and works with Tweets, which usually have a different structure than other online user comments due to character limitation.