Benchmarking sentiment analysis methods for large-scale texts: A case for using continuum-scored words and word shift graphs.
Table S1: Ranked results of sentiment dictionary performance on individual tweets from STS-Gold dataset (Saif, 2013). We report the percentage of tweets for which each dictionary contains at least 1 entry, the F1 score on those tweets, and the overall classification F1 score. The calibrated F1 score tunes the decision threshold between positive and negative tweets with a random 10% training sample.
['Rank', 'Dictionary', '% tweets scored', 'F1 of tweets scored', 'Calibrated F1', 'Overall F1']
As all methods must, dictionary-based “bag-of-words” approaches suffer from various drawbacks, and three are worth stating up front. First, they are only applicable to corpora of sufficient size, well beyond that of a single sentence (widespread usage in this misplaced fashion does not suffice as a counterargument). We directly verify this assertion on individual Tweets, finding that some dictionaries perform admirably, however the average (median) F1-score However, in practice the domains and topics to which sentiment analysis are applied are highly varied, such that training to a high degree of specificity for a single corpus may not be practical and, from a scientific standpoint, will severely constrain attempts to detect and understand universal patterns. Third: words may be evaluated out of context or with the wrong meaning. A simple example is the word “new” occurring frequently when evaluating articles in the New York Times. This kind of contextual error is something we can readily identify and correct for through word shift graphs, but would remain hidden to nonlinear learning methods without new training.