Information-Weighted Neural Cache Language Models for ASR
Table 1: Validation and test perplexities for LSTM LMs with a cache trained on WikiText-2, along with their optimal hyperparameters (if applicable). The column λ / γ contains the optimal interpolation weight λ for models with linear interpolation and the optimal scale for the IW γ for models with IW interpolation.
['[BOLD] Model', '[BOLD] Size', '[BOLD] Interp.', '[BOLD] IW select.', '[BOLD] Regular cache  [BOLD] Valid', '[BOLD] Regular cache  [BOLD] Test', '[BOLD] Regular cache  [ITALIC] λ\xa0/\xa0 [ITALIC] γ', '[BOLD] Regular cache  [ITALIC] ϕ', '[BOLD] Regular cache  [ITALIC] α', '[BOLD] Neural cache  [BOLD] Valid', '[BOLD] Neural cache  [BOLD] Test', '[BOLD] Neural cache  [ITALIC] λ\xa0/\xa0 [ITALIC] γ', '[BOLD] Neural cache  [ITALIC] ϕ', '[BOLD] Neural cache  [ITALIC] θ']
Due to implementation requirements, we have to train 4 different models: one with a non-selective cache of size 100, one with a selective cache of size 100 and the same options for a cache size of 2000. The baseline perplexities for LSTMs without cache of these four models do not differ much, but we report them all for the sake of completeness (‘Our baseline’ in the table). All perplexity results shown below the same baseline can thus be attributed solely to the cache component since they are obtained with the same trained model.