Learning to Compute Word Embeddings On the Fly
Table 3: Results on 1% of OBW and full OBW training set. We report 2 variants of perplexities on the test set. The experiments where coverage is restricted are indicated by (R). See Section 4.3 for explanations.
['model', '1% PPL', '1% PPL after OOV', '100% PPL', '100% PPL after OOV']
The training was early-stopped using a development set. Similarly to our other experiments, using external information to compute embeddings of unknown words helps in all cases. We observe a significant gain even for dict1, which is remarkable as this model has the same architecture and parameters as the baseline. We note that lemma+lowercase performs worse than any model with the dictionary, which suggests that dictionary definitions are used in a non-trivial way. Adding spelling consistently helps more than adding dictionary definitions. Using both dictionary and spelling is consistently slightly better than using just spelling, and the improvement is more pronounced in the restricted setting. Using GloVe embeddings results in the best perplexity. Switching to the full training set shrank all the gaps.