Definition Modeling: Learning to define word embeddings in natural language
Table 7: Percentage of times a definition is manually ranked in each position (@k), and average rank (Avg).
['Choices', '@1', '@2', '@3', '@4', 'Avg']
First, we perform a qualitative evaluation of the models’ output by asking 6 participants to rank a set of definitions of 50 words sampled from the test set. For each word w, we provide in random order: a ground-truth definition for w (Dictionary), a ground-truth definition of the word w′ whose embedding is nearest to that of w (NE), the standard language model (Seed*), and our full system (S+G+CH+HE*). Inter-annotator agreement was strong (almost all inter-annotator correlations were above 0.6). The advantage of S+G+CH+HE* over Seed* is statistically significant (p<0.002, t-test), and the difference between S+G+CH+HE* is and NE is borderline significant (p<0.06, t-test).