HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
Table 11: Results in the graded LE task over all HyperLex concept pairs obtained by the sets of most prominent LE models available in the literature (see Sect. 6.1-Sect. 6.5). Setup 1 and Setup 2 refer to different training setups for DEMs and SLQS. All results are Spearman’s ρ correlation scores. IAA ρ scores are provided to quantify the upper bound for the graded LE task.
['Model', '[BOLD] Setup 1', '[BOLD] Setup 2']
S7SS2SSS0Px1 Experiment I: Ungraded LE Approaches In the first batch of experiments, we evaluate a series of state-of-the-art traditional LE modelling aproaches in the graded LE task on the entire HyperLex evaluation set. The models are described in Sect. A summary of the results is provided in Tab. Comparing model scores with the inter-annotator agreements suggests that the graded LE task, although well-defined and understandable by average native speakers, poses a challenge for current ungraded LE models. The absolute difference in scores between human and system performance indicates that there is vast room for improvement in future work. The gap also illustrates the increased difficulty of the graded LE task compared to previous ungraded LE evaluations (see also Exp. IV). For instance, the best unsupervised LE directionality and detection models from Tab. The results also reveal that visual approaches are competitive to purely textual distributional ones. In Tab. Varying the α parameter leads to even better results, e.g., the Vis-ID model scores ρ=0.229 and Vis-Cent scores ρ=0.228 with α=1. This finding supports recent trends in multi-modal semantics and calls for more expressive multi-modal LE models as discussed previously by Kiela et al. \shortciteKiela:2015acl.