Benchmarking sentiment analysis methods for large-scale texts: A case for using continuum-scored words and word shift graphs.
Table S4: Ranked performance of dictionaries on the Movie Review corpus.
['Rank', 'Title', '% Scored', 'F1 Trained', 'F1 Untrained']
Classifying single reviews as positive or negative, the F1-scores are: LabMT .63, ANEW .36, LIWC .53, MPQA .66, OL .71, and WK .34 We roughly confirm the rule-of-thumb that 10,000 words are enough to score with a dictionary confidently, with all dictionaries except MPQA and ANEW achieving 90% accuracy with this many words. We sample the number of reviews evenly in log space, generating sets of reviews with average word counts of 4550, 6500, 9750, 16250, and 26000 words. Specifically, the number of reviews necessary to achieve 90% accuracy is 15 reviews (9750 words) for LabMT, 100 reviews (65000 words) for ANEW, 10 reviews (6500 words) for LIWC, 10 reviews (6500 words) for MPQA, 7 reviews (4550 words) for OL, and 25 reviews (16250 words) for WK.