Definition Modeling: Learning to define word embeddings in natural language
Table 5: Equally-weighted BLEU scores for up to 4-grams, on definitions evaluated using different reference dictionaries (results are not comparable between columns).
['Model', 'GC', 'WN', 'Avg', 'Merged']
AVG and Merge in the table are two ways of aggregating the BLEU score. AVG averages the BLEU scores by using each dictionary as the ground truth. The Merge computes score by using union of the two dictionaries. First, we can see that the baselines have low BLEU scores when evaluated on definitions from the other dictionary (Inter and NE-). This shows that different dictionaries use different styles. However, despite the fact that our best model S+G+CH is unaware of which dictionary it is evaluated against, it generates definitions that strike a balance between both dictionaries, and achieves higher BLEU scores overall. As in the earlier experiments, the Gated model improves the most over the Seed model. In addition, the affixes further improves the performance while the contribution of the hypernym embeddings is unclear on this task. We believe the other error types stem more from semantic gaps in the embeddings than from limitations in the definition model. Our reasons for placing the blame on the embeddings rather than the definition model itself are twofold. First, we perform an ablation study in which we train S+G+CH using randomized embeddings, rather than the learned Word2Vec ones. Secondly, the error types (4) - (6) are plausible shortcomings of embeddings, some of which have been reported in the literature. For example, the word “captain” often appears near the word “ship”, but the role (as a leader) is frequently implicit.