SemEval-2013 Task 2: Sentiment Analysis in Twitter
Table 4: Bounds for datasets in subtasks A and B.
['[EMPTY]', '[BOLD] A  [BOLD] Lower', '[BOLD] A  [BOLD] Avg.', '[BOLD] A  [BOLD] Upper', '[BOLD] B  [BOLD] Avg.']
We also experimented with combining annotations by computing the union of the sentences, and taking the sentence of the worker who annotated the most hits, but we found that these methods were not as accurate. This gives a good indication about how well we can expect the systems to perform. For example, even if we used the best annotator each time, it would still not be possible to get perfect accuracy. For subtask B, the polarity of the entire sentence was determined based on the majority of the labels. If there was a tie, the sentence was discarded. In order to reduce the number of sentences lost, we combined the objective and the neutral labels, which Turkers tended to mix up. Since the polarity is chosen based on the majority, the upper bound is 100%.