Large Memory Layers with Product Keys
Table 1: Test perplexity for models with and without memory. PKM models with 12 layers outperforms 24-layer models of same dimensionality. Bold refers to models optimizing performance for a given dimension.
['Dimension N memories', '1024 0', '1024 1', '1024 2', '1024 3', '1600 0', '1600 1']
We observe that increasing either the dimensionality or the number of layers leads to significant perplexity improvements in all the models. However, adding a memory to the model is more beneficial than increasing the number of layers; for instance, a model with a single memory and 12 layers outperforms a memoryless model with the same hidden dimension and 24 layers, both when the number of hidden units is 1024 and 1600. Adding 2 or 3 memory layers further improves performance.