Direct Output Connection for a High-Rank Language Model
Table 6: Perplexities of our implementations and re-runs on the PTB dataset. We set the non-monotone interval to 60. † represents results obtained by original implementations with identical hyperparameters except for non-monotone interval. ‡ indicates the result obtained by our AWD-LSTM-MoS implementation with identical dropout rates as AWD-LSTM-DOC. For (fin), we repeated fine-tuning until convergence.
['Model', 'Valid', 'Test']
For AWD-LSTM-MoS, we trained our implementation with the same dropout rates as AWD-LSTM-DOC for a fair comparison. AWD-LSTM-DOC outperformed both the original AWD-LSTM-MoS and our implementation. In other words, DOC outperformed MoS. Since the averaged SGD uses the averaged parameters from each update step, the parameters of the early steps are harmful to the final parameters. Therefore, when the model converges, recent studies and ours eliminate the history of and then retrains the model. \newcitemerityRegOpt referred to this retraining process as fine-tuning. Although most previous studies only conducted fine-tuning once, \newcitefraternal argued that two fine-tunings provided additional improvement. Thus, we repeated fine-tuning until we achieved no more improvements in the validation data.