Attentive Mimicking:Better Word Embeddings by Attending to Informative Contexts
Table 4: Spearman’s ρ for the Chimeras task given 2, 4 and 6 context sentences for the made-up word
['[BOLD] model', '2 sent.', '4 sent.', '6 sent.']
The Chimeras (CHIMERA) dataset Lazaridou et al. CHIMERA provides only six contexts for each made-up word, so it is not ideal for evaluating our model. Nonetheless, we can still use it to analyze the difference of FCM (no attention) and AM (using attention). As the surface-form of the made-up words was constructed randomly and thus carries no meaning at all, we restrict ourselves to the context parts of FCM and AM (referred to as FCM-ctx and AM-ctx). We do not report results for Mimick as its representations for novel words are entirely based on their surface form. While AM performs worse than previous methods for 2–4 sentences, it drastically improves over the best result currently published for 6 sentences. Again, context attention consistently improves results: AM-ctx performs better than FCM-ctx, regardless of the number of contexts. Since A La Carte Khodak et al. , the method performing best for 2–4 contexts, is conceptually similar to FCM, it most likely would similarly benefit from context attention.