Detecting Offensive Language in TweetsUsing Deep Learning
Table 3: Evaluation Results
['[BOLD] Approach', '[BOLD] Characteristics', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F-Score']
We now present the most interesting results from our experiments. For the evaluation we used standard metrics for classification accuracy, suitable for studying problems such as sentiment analysis. In particular we used Precision and Recall, with the former calculated as the ratio of the number of tweets correctly classified to a given class over the total number of tweets classified to that class, while the latter measures the ratio of messages correctly classified to a given class over the number of messages from that class. Additionally, the F-score is the harmonic mean of precision and recall , expressed as F=2⋅P⋅RP+R. For our particular case with three classes, P, R and F are computed for each class separately, with the final F value derived as the weighted mean of the separate F-scores: F=FN⋅N+FR⋅R+FS⋅SN+R+S ; recall that N=10889, S=3166 and R=1943. Note that the performance numbers P,R and F of the other state of the art approaches are based on the authors’ reported data in the cited works. Additionally, we report the performance of each individual LSTM classifier as if used alone over the same data (that is, without the ensemble logic). The F-score for our proposed approaches shown in the last column, is the weighted average value over the 3 classes (Neutral,Sexism,Racism). Moreover, all the reported values are average values produced for a number of runs of the same tested scheme over the same data. We clearly see that the models converge. For the final run the F-score has standard deviation value not larger than 0.001, for all classifiers.