SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods
Tabela 6: Mean Rank Table for All Datasets
['3-Classes  [BOLD] Pos', '3-Classes  [BOLD] Method', '3-Classes  [BOLD] Mean Rank', '2-Classes  [BOLD] Pos', '2-Classes  [BOLD] Method', '2-Classes  [BOLD] Mean Rank', '2-Classes  [BOLD] Coverage (%)']
Umigon, the best method in five Twitter datasets, felt to the eighteenth place in the Comments_NYT dataset. We can also note the lower Macro-F1 values for some methods like Emoticons are due to the high number of sentences without emoticons in the datasets. Methods like Emoticons DS and PANAS tend do classify only a small part of instances as neutral and also presented a poor performance in the 3-class experiments. Methods like SenticNet and LIWC were not originally developed for detecting neutral sentences and also achieved low values of Macro-F1. However, they also do not appear among the best methods in the 2-class experiments, which is the task they were originally designed for. The elements are sorted by the overall mean rank each method achieved based on Macro-F1 for all datasets. The top nine methods based on Macro-F1 for the 2-class experiments are: SentiStrength, Sentiment140, Semantria, OpinionLexicon, LIWC15, SO-CAL, AFINN and VADER and Umigon. With the exception of Sentistrength, replaced by Pattern.en, the other eight methods produce the best results across several datasets for both, 2 and 3-class tasks. These methods would be preferable in situations in which any sort of preliminary evaluation is not possible to be done. The mean rank for 2-class experiments is accompanied by the coverage metric, which is very important to avoid misinterpretation of the results. Observe that SentiStrength and Sentiment140 exhibited the best mean ranks for these experiments, however both present very low coverage, around 30% and 40%, a very poor result compared with Semantria and OpinionLexicon that achieved a worse mean rank (4,61 and 6,62 respectively) but an expressive better coverage, above 60%. Note also that SentiStrength and Sentiment140 present poor results in the 3-class experiments which can be explained by their bias to the neutral class as mentioned before.