Improved Language Modeling by Decoding the Past
Table 7: Ablation experiments on the PTB and WT2 validation sets.
['[BOLD] Model', '[BOLD] PTB Valid', '[BOLD] PTB Test', '[BOLD] WT2 Valid', '[BOLD] WT2 Test']
We perform a set of ablation experiments on the best AWD-LSTM+PDR models for PTB and WT2 to understand the relative contribution of PDR and the other regularizations used in the model. In both cases, PDR has a significant effect in decreasing the validation set performance, albeit lesser than the other forms of regularization. This is not surprising as PDR does not influence the LSTM directly.