Crowdsourcing for Beyond Polarity Sentiment AnalysisA Pure Emotion Lexicon
Table 4: Fleiss k for different numbers of total annotations
['[BOLD] Total annotations', '[BOLD] Subclass  [ITALIC] k', '[BOLD] Emotional  [ITALIC] k']
As the number of annotators rises, the number of possible agreement permutations increases but the number of major annotated subclasses decreases. More annotators have a positive effect in both k and certainty of classification.