Attending the Emotions to Detect Online Abusive Language
Table 3: Classification results in terms of F1 score for the negative/offensive class and macro F1. The results of our proposed model are significantly better than Sam’17 under the Mcnemar significance test.
['[BOLD] Model', '[BOLD] Model', '[BOLD] Curious Cat F1', '[BOLD] Curious Cat Macro F1', '[BOLD] ask.fm F1', '[BOLD] ask.fm Macro F1', '[BOLD] Kaggle F1', '[BOLD] Kaggle Macro F1', '[BOLD] Wikipedia F1', '[BOLD] Wikipedia Macro F1']
Overall, compared to \newcitesamghabadi2017detecting, all of the improvements with our model are statistically significant under the Mcnemar significance test (p−value<0.001) for all of the resources. Our model outperforms the-state-of-art for the ask.fm dataset by about 1.5% with respect to the macro F1. The obtained AUC score for this dataset with our model is 0.913, which shows an improvement of 7%. Only for the Kaggle dataset, the performance of the EA model is slightly worse than the RA. We believe that this is due to the diverse nature of data that we have in different datasets. For the Wikipedia data, we observe that the CNN module obtains the best results (around 1.5% better than the state-of-the-art F1 score); however, the difference (0.1%) between the performance of this model and our proposed stacked CNN-BiLSTM is not significant based on the Mcnemar test. We could also observe that concatenating emoji vectors with the final hidden representation clearly boosts the performance of the system, which confirms our assumption that using the emotional information behind the text benefits the model.