Information-Weighted Neural Cache Language Models for ASR
Table 2: Perplexities on the validation set of WSJ and WERs after rescoring the 1000-best lists of the validation and test set. All cache models have a cache size of 100. ‘poi’ = probability of improvement [17] of the cache model with respect to its baseline. The 5 last columns contain the hyperparameter values optimized on the validation set by rescoring, which are also used for rescoring the test set. The column λ / γ contains the optimal λ for models with linear interpolation and the optimal γ for models with IW interpolation. We do not include the decay values since exponential decay did not improve the results. Oracle WER for the 1000-best lists is 2.18 (validation) and 1.21 (test).
['[BOLD] Model', '[BOLD] Valid PPL', '[BOLD] Valid WER (poi)', '[BOLD] Test WER (poi)', '[ITALIC] λ /  [ITALIC] γ', '[ITALIC] ϕ', '[ITALIC] λLSTM', '[ITALIC] β', '[ITALIC] ω']
Similarly as for WikiText, we report the results for two different baseline models, a selective one and a non-selective one. Notice that all results within the upper part and respectively the lower part of the table are obtained with the same trained model. Thus, differences in performance can be attributed to the cache parts solely and not to different initialization settings that might lead to better convergence of the model during training. As a sanity check, we tested whether the differences in WER between the two baseline models were significant: Thus, we conclude that the difference between the baselines is not significant. We only test models with a cache size of 100, since preliminary perplexity results with a larger cache size did not give substantial improvements with respect to a cache of 100 for this dataset.