SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods
Tabela 5: 3-classes experiments results with 4 datasets
['[BOLD] Dataset', '[BOLD] Method', '[BOLD] Accur.', '[BOLD] Posit. Sentiment  [BOLD] P', '[BOLD] Posit. Sentiment  [BOLD] R', '[BOLD] Posit. Sentiment  [BOLD] F1', '[BOLD] Negat. Sentiment  [BOLD] P', '[BOLD] Negat. Sentiment  [BOLD] R', '[BOLD] Negat. Sentiment  [BOLD] F1', '[BOLD] Neut. Sentiment  [BOLD] P', '[BOLD] Neut. Sentiment  [BOLD] R', '[BOLD] Neut. Sentiment  [BOLD] F1', '[BOLD] MacroF1']
We start the analysis of our experiments by comparing the results of all previously discussed metrics for all datasets. For simplicity, we choose to discuss results only for these datasets as they come from different sources and help us to illustrate the main findings from our analysis. Results for all the other datasets are presented in the Additional File 1. There are many interesting observations we can make from these results, summarized next. Umigon, the best method in five Twitter datasets, felt to the eighteenth place in the Comments_NYT dataset. We can also note the lower Macro-F1 values for some methods like Emoticons are due to the high number of sentences without emoticons in the datasets. Methods like Emoticons DS and PANAS tend do classify only a small part of instances as neutral and also presented a poor performance in the 3-class experiments. Methods like SenticNet and LIWC were not originally developed for detecting neutral sentences and also achieved low values of Macro-F1. However, they also do not appear among the best methods in the 2-class experiments, which is the task they were originally designed for. Existing methods let space for improvements: We can note that the performance of the evaluated methods are ok, but there is a lot of space for improvements. Considering that we are looking at the performance of the best methods out of 24 unsupervised tools, these numbers suggest that current sentence-level sentiment analysis methods still let a lot of space for improvements. Additionally, we also noted that the best method for each dataset varies considerably from one dataset to another. This might indicate that each method complements the others in different ways.