Attentive Mimicking:Better Word Embeddings by Attending to Informative Contexts
Table 1: Average cosine similarities for the VecMap evaluation, scaled by a factor of 100. †: Downsampled words were included in the training set.
['[BOLD] model', '[BOLD] number of occurrences 1', '[BOLD] number of occurrences 2', '[BOLD] number of occurrences 4', '[BOLD] number of occurrences 8', '[BOLD] number of occurrences 16', '[BOLD] number of occurrences 32', '[BOLD] number of occurrences 64', '[BOLD] number of occurrences 128']
As baselines, we train skipgram and fastText on the downsampled corpus. We then train Mimick Pinter et al. We also try a variant where the downsampled words are included in the training set (i.e., the mimicking models explicitly learn to reproduce their skipgram embeddings). This allows the model to learn representations of those words not completely from scratch, but to also make use of their original embeddings. Accordingly, we expect this variant to only be helpful if a word is not too rare, i.e. its original embedding is already of decent quality. The comparably poor performance of Mimick is consistent with the observation of Pinter et al. Given four or more contexts, AM leads to consistent improvements over FCM. The variants that include downsampled words during training (†) still outperform skipgram for 32 and more observations, but perform worse than the default models for less frequent words.