Learning Semantic Relatedness from Human Feedback Using Metric Learning
Table 1: Overview of all Human Intution Datasets (HIDs). For each HID, we give the number of word pairs, the number of unique words and the number of judgments per word pair. Also, for each embedding dataset, we give the number of matchable pairs, where both words are present in the dataset’s vocabulary.
['dataset', 'pairs', 'words', 'matches BibSonomy', 'matches Delicious', 'matches WikiNav', 'matches WikiGloVe']
As a gold standard for semantic relatedness as it is perceived by humans, we use several datasets with human-generated relatedness scores for word pairs, so called human intuition datasets (HIDs). They will provide training as well as test data. In the following, we will describe all used HIDs briefly. Integrating Different Amounts of User Feedback. It can be seen that in the random sampling experiments, the information in the MEN dataset is generally best suited to learn semantic relatedness. Using crowdsourcing, each human worker was shown two pairs of words and had to determine which of both pairs is more related. The higher a word pair in MEN is rated, the more often it was considered more related than the other one that was given. Our approach exploits very similar constraints for learning. Keeping this explanation in mind, we are able to give a recommendation on how to gather human feedback in order to learn semantic relatedness with our method. On all other embedding datasets, where MEN is very useful to learn the metric, the pair overlap is notably higher. Throughout all settings in this experiment, we observed that using more data results in better relatedness scores, if there was a positive effect.