Improving Neural Language Models with a Continuous Cache
(b) lambada
['Model', 'Dev', 'Ctrl']
Finally, we report experiments carried on the lambada dataset, introduced by Paperno et al. This is a dataset of short passages extracted from novels. The goal is to predict the last word of the excerpt. This dataset was built so that human subjects solve the task perfectly when given the full context (approx. 4.6 sentences), but fail to do so when only given the sentence with the target word. Thus, most state-of-the-art language models fail on this dataset. The lambada training set contains approximately 200M tokens and has a vocabulary size of 93,215. Adding a neural cache model to the LSTM baseline strongly improves the performance on the lambada dataset. This is due to the fact that more than 83% of passages of the development set include the target word, while this is true for only 14% of the control set. Ideally, a model should have strong results on both sets. One possible generalization of our model would be to adapt the interpolation parameter based on the current vector representation of the history ht.