SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods
Tabela 4: 2-classes experiments results with 4 datasets
['[BOLD] Dataset', '[BOLD] Method', '[BOLD] Accur.', '[BOLD] Posit. Sentiment  [BOLD] P', '[BOLD] Posit. Sentiment  [BOLD] R', '[BOLD] Posit. Sentiment  [BOLD] F1', '[BOLD] Negat. Sentiment  [BOLD] P', '[BOLD] Negat. Sentiment  [BOLD] R', '[BOLD] Negat. Sentiment  [BOLD] F1', '[BOLD] MacroF1', '[BOLD] Coverage']
We start the analysis of our experiments by comparing the results of all previously discussed metrics for all datasets. For simplicity, we choose to discuss results only for these datasets as they come from different sources and help us to illustrate the main findings from our analysis. Results for all the other datasets are presented in the Additional File 1. There are many interesting observations we can make from these results, summarized next. Methods prediction performance varies considerably from one dataset to another: First, we note the same social media text can be interpreted very differently depending on the choice of a sentiment method. Overall, we note that all the methods yielded with large variations across the different datasets. Although the first two datasets contain tweets, they belong to different contexts, which affects the performance of some methods like VADER. Another important aspect to be analyzed in this Table is the coverage. Although SentiStrength has presented good Macro-F1 values, its coverage is usually low as this method tends to classify a high number of instances as neutral. Another expected result is the good Macro-F1 values obtained by Emoticons, specially in the Twitter datasets. It is important to highlight that, in spite of achieving high accuracy and Macro-F1, the coverage of many methods, such as PANAS, VADER, and SentiStrength, is low (e.g. below 30%) as they only infer the polarity of part of the input sentences. Thus, the choice of a sentiment analysis is highly dependent on the data and application, suggesting that researchers and practitioners need to take into account this tradeoff between prediction performance and coverage. Existing methods let space for improvements: We can note that the performance of the evaluated methods are ok, but there is a lot of space for improvements. Considering that we are looking at the performance of the best methods out of 24 unsupervised tools, these numbers suggest that current sentence-level sentiment analysis methods still let a lot of space for improvements. Additionally, we also noted that the best method for each dataset varies considerably from one dataset to another. This might indicate that each method complements the others in different ways.