Large Memory Layers with Product Keys
Table 2: Perplexity and memory usage for different memory sizes, with and without BatchNorm. Adding a batch normalization layer in the query network encourages the model to use more keys. This is not necessary for small memories of size 16k and 65k where the usage is already close to 100% without batch normalization, but for memories of size 147k of more, batch normalization improves the memory usage significantly, along with the perplexity.
['Memory size BatchNorm', '16k No', '16k Yes', '65k No', '65k Yes', '147k No', '147k Yes', '262k No', '262k Yes', '590k No', '590k Yes', '1M No', '1M Yes']
We train transformer models with memories of size |K|=|C|×|C′|, with |C′|=|C| and |C|∈{128,256,384,512,768,1024}. A model with a memory size of 16k obtains a perplexity of 22.7972. Increasing the size to 1M decreases the perplexity down to 17.9819 while leaving the inference time unchanged. The dominant factor for inference time is the number of accessed memory values, which is governed by the number of memory heads and the parameter k, but not the memory size.