Generalization through Memorization: Nearest Neighbor Language Models
Table 3: Experimental results on Wiki-3B. The model trained on 100M tokens is augmented with a datastore that contains about 3B training examples, outperforming the vanilla LM trained on the entire Wiki-3B training set.
['[BOLD] Training Data', '[BOLD] Datastore', '[BOLD] Perplexity (↓) Dev', '[BOLD] Perplexity (↓) Test']
However, adding nearest neighbors retrieval over those 3B examples to the model trained on 100M tokens improves perplexity from 19.59 to 13.73; i.e. retrieving nearest neighbors from the corpus outperforms training on it. This result suggests that rather than training language models on ever larger datasets, we can use smaller datasets to learn representations and augment them with kNN-LM over a large corpus.