Attentive Mimicking:Better Word Embeddings by Attending to Informative Contexts
Table 5: Context sentences and corresponding attention weights for the made-up word “petfel”
['sentence ∙', 'sentence i doubt if we ll ever hear a man play a  [BOLD] petfel like that again', '[ITALIC] ρ 0.19']
The model attends most to sentences (2) and (4); consistently, the embeddings obtained from those sentences are very similar. Furthermore, of all four sentences, these two are the ones best suited for a simple averaging model as they contain informative, frequent words like “instrument”, “chimes” and “music”.