HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
Table 6: Graded LE scores for instances of several prominent taxonomical categories/classes represented in HyperLex (i.e., the categories are the word Y in each (X,Y,s) graded LE triplet).
['type-of cat', '[BOLD] animal 10.0', 'sandwich', '[BOLD] food 10.0', 'rose', '[BOLD] plant 9.75', 'basketball', '[BOLD] sport 10.0', 'girl', '[BOLD] person 9.85', 'car', '[BOLD] vehicle 10.0']
In the first qualitative analysis, we aim to investigate the straightforward question: are some concepts really more (proto)typical of semantically broader higher-level classes? Several examples of prominent high-level taxonomical categories along with LE scores are shown in Tab. We might draw several preliminary insights based on the presented lists. There is an evident prototyping effect present in human judgements: concepts such as cat, monkey or cow are more typical instances of the class animal than the more peculiar instances such as mongoose or snail according to HyperLex annotators. Instances of the class sport also seem to be sorted accordingly, as higher scores are assigned to arguably more prototypical sports such as basketball, volleyball or soccer, and less protypical sports such as racquetball or wrestling are assigned lower scores. Nonetheless, the majority of hyp-N pairs (X,animal) or (X,sport), where X is a hyponym of animal/sport according to WN, are indeed assigned reasonably high graded LE scores. It suggests that humans are able to: (1) judge the LE relation consistently and decide that a concept indeed stands in a type-of relation with another concept, and (2) grade the LE relation by assigning more strength to more prototypical class instances. Similar patterns are visible with other class instances from Tab. We also observe the same effect with verbs, e.g., (drift, move, 8.58), (hustle, move, 7.67), (tow, move, 7.37), (wag, move, 6.80), (unload, move, 6.22). Further, nominal concepts higher in the WN hierarchy typically refer to semantically very broad but well-defined categories such as animal, food, vehicle, or appliance (see again Tab. Semantically more specific instances of such concepts are easier to judge as true hyponyms (using the ungraded LE terminology), which also reflects in higher LE ratings for such instances. However, gradience effects are clearly visible even for pairs with longer WN distances (Tab.