Generalization through Memorization: Nearest Neighbor Language Models
Table 5: Wikitext-103 validation results using different states from the final layer of the LM as the representation function f(⋅) for keys and queries. We retrieve k=1024 neighbors and λ is tuned for each.
['[BOLD] Key Type', '[BOLD] Dev ppl. (↓)']
, we extract a representation of context c using an intermediate state of the LM f(c). While all the instantiations of f we tried are helpful, we achieved the largest improvement by using the input to the final layer’s feedforward network. We also observe that normalized representations (i.e. taken immediately after the layer norm) perform better. Repeating the experiment on the second-last transformer layer showed similar trends with slightly worse results (not shown), suggesting that the feedforward layer might be focusing more on the prediction problem, while the onus of representing the input falls more on the self-attention layer.