HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
Table 16: The effects of lexical memorisation on the output of regression models when dealing with typical hypernymy concepts higher in the taxonomy (e.g., animal, plant): HyperLex denotes the score assigned to the pair by humans in HyperLex, while ols and ridge refer to the predicted output of the two tested regression models. We use SGNS-DEPS embeddings with concatenation (→X⊕→Y, see Fig. 8), while similar trends are observed with other sets of vectors and feature transformations.
['Concept pair', 'HyperLex', 'ols', 'ridge']
Besides operating with a smaller training set in a lexical split, the finding is also explained by the effect of lexical memorisation with a random split: if high scores are systematically assigned to training pairs (X1, animal) or (X2, appliance), the model will simply memorise that each pair (Y1, animal) or (Y2, appliance) should be assigned a high score during inference. The impact of lexical memorisation is illustrated by Tab. : the regression models assign high scores even to clear negatives such as (plant, animal). However, the effect of lexical memorisation also partially explains the improved performance of all regression models over Unsupervised baselines for a random split, as many (Xt, animal) pairs are indeed assigned high scores in the test set.