Table 3: Distribution of labels in phrase-bank for 4 subsets formed based on the strength of majority agreement. Each sentence has 5 to 8 overlapping annotations, which have been used to determine the degree of agreement.
['[EMPTY]', '%Negative', '%Neutral', '%Positive', 'Count']
Given the large number of overlapping annotations (5 to 8 annotations per sentence), there are several ways to define a majority vote based gold standard. To provide an objective comparison, we have formed 4 alternative reference datasets based on the strength of majority agreement: (i) sentences with 100% agreement; (ii) sentences with more than 75% agreement; (iii) sentences with more than 66% agreement; and (iv) sentences with more than 50% agreement (simple majority based Gold standard). In the experiments, all datasets have been used to evaluate the performance of the LPS models with respect to a number of competing baseline models. All of the datasets are quite similar in terms of label distribution.