Improved Language Modeling by Decoding the Past
Table 6: Validation perplexities for AWD-LSTM without any regularization and with only PDR.
['[EMPTY]', 'PTB Valid', 'WT2 Valid']
To verify that indeed PDR can act as a form of regularization, we perform the following experiment. We take the models for PTB and WT2 and turn off all dropouts and regularization and compare its performance with only PDR turned on. The model with only PDR turned on achieves 2.4 and 5.1 better validation perplexity on PTB and WT2 as compared to the model without any regularization. Thus, biasing the LSTM by decoding the distribution of past tokens from the predicted next-token distribution can indeed act as a regularizer leading to better generalization performance.