Detecting and ordering adjectival scalemates Paper presented at MAPLEX 2015, February 9-10, Yamagata, Japan (http://lang.cs.tut.ac.jp/maplex2015/)
Table 3: Results for the evaluation using only pairs for which the MPQA and WKB data agreed on the ordering. Scores are given for the unfiltered data. Filtering generally had a negative effect on the score of about one percent.
['Method', '# Items', 'Score']
The WKB evaluation yields slightly lower scores than those obtained with the MPQA dataset (56–60%). But how reliable are those scores? To find out, we took the raw scalemates and compared the orderings predicted by the MPQA and WKB datasets. It turns out that they agree on only 62% of the orderings. This is a surprisingly low number, which casts doubt on the value of these data sets as an individual evaluation metric for adjectival scales. We made the evaluation more robust by combining the two evaluation sets, using only those pairs for which both sets agree on the order.