Large Memory Layers with Product Keys
Table 4: Perplexity, memory usage and inference speed with product keys and regular keys. Models with product keys have a much better usage than models that represent keys by a flat matrix, and obtain a better perplexity. They also have significantly less parameters and are dramatically faster to run. The speed is measured at inference, in thousands of words per second (w/s). For models with more than 262k memory slots, we only report the inference time. We observe that with product keys, the memory size do not impact the inference time.
['Memory size Product Keys', '16k No', '16k Yes', '65k No', '65k Yes', '147k No', '147k Yes', '262k No', '262k Yes', '590k No', '590k Yes', '1M No', '1M Yes']
As a result, product keys contain |C| times less parameters than keys represented by a full matrix. In the second case, searching the nearest keys boils down to a liner index search at each iteration, which is computationally very expensive. As a result, we only report results for memories of size 16k, 65k, 147k, as experiments with a flat index on larger memories takes an unreasonable amount of time to converge. We can see that models with product keys are not only faster but they have also a much better memory usage, and consequently obtain a better perplexity.