Large Memory Layers with Product Keys
Table 3: Perplexity and memory usage for different memory positions in a transformer with 6 layers. Adding a memory in positions 4 or 5 maximizes the performance (layer 1 is the worst).
['Position', '1', '2', '3', '4', '5', '6']
In this experiment we insert the memory at different levels in the transformer, to see where it is the most beneficial. Putting memory at layer 1 (after the input token embeddings) gives the worst performance. When the memory is inserted in layer 6, it is located right before the softmax output, the model has only one linear layer to process the information read from the memory. The best position to insert the memory is at an intermediate layer. We surmise that effective use of the memory requires operating in a more abstract feature space than the input and that it is important to have some layers on the top of the memory to further process and aggregate information from every location.