Detecting and ordering adjectival scalemates Paper presented at MAPLEX 2015, February 9-10, Yamagata, Japan (http://lang.cs.tut.ac.jp/maplex2015/)
Table 4: Results for our new evaluation set and our algorithm on the evaluation data provided by Christopher Potts (p.c.). The columns correspond to the level of agreement between participants. I.e. how many participants (out of 10) agreed on the first adjective being either stronger or weaker than the second. Making this requirement more strict reduces the amount of test items that we could use, but increases the precision of our evaluation set and our algorithm.
['Our proposed data set Agreement', 'Our proposed data set 6', 'Our proposed data set 7', 'Our proposed data set 8', 'Our proposed data set 9', 'Our proposed data set 10']
After we finished our data-analysis, Christopher Potts (p.c.) shared the results of an online experiment carried out on Amazon’s Mechanical Turk. Participants were shown a set of adjective pairs, and asked for each pair to judge whether the first adjective is stronger, weaker or as strong as the second adjective. This is exactly the kind of data we need to evaluate the order of automatically identified scales. We observe that the combination of the MPQA and WKB data provides a reasonable estimate of the correct ordering of adjectival scales, but our algorithm does much better on the elicited data than on our proposed evaluation set.