A naive but challenging baseline for mapping distributional to model-theoretic semantic spaces
Table 1: All the presented results are averaged over 1000 runs, except for the results of [Herbelot and Vecchi2015]) in the last row. PLSR stands for partial least squares regression, NN for nearest neighbor, ppdb for the Paraphrase Database [Ganitkevitch et al.2013]. There are two ways to compute the mode: either taking the mode of the means of the 3 annotations (mode), or the mode for all annotations (true-mode). QMR has 3 potentially different annotations for each concept-feature pair, while AD has 3 only one annotation for each concept-feature pair: as a result, mode and true-mode have similar results for AD, but potentially different results for QMR. For each run, a train/test split was randomly chosen (60 training samples for AD, 400 for QMR, in order to have the same number of training samples as in [Herbelot and Vecchi2015]’s Table 2).
['[EMPTY]', 'AD Min', 'AD Average', 'AD Max', 'QMR Min', 'QMR Average', 'QMR Max']
We could reproduce the results for the QMR dataset using PLSR and word2vec embeddings (0.346 Our experiments’ results are averaged over 1000 runs, and for each run For the AD dataset, our worst run achieved 0.435, and our best run achieved 0.713, which emphasizes the lack of robustness of the results with respect to the train/test split. The variability is much lower for the QMR dataset (min: 0.244; max: 0.407), which is expected since QMR is significantly larger than AD.