Direct Output Connection for a High-Rank Language Model
Table 2: Hyperparameters used for training DOC.
['Hyperparameter', 'PTB', 'WikiText-2']
Our implementation is based on the averaged stochastic gradient descent Weight-Dropped LSTM (AWD-LSTM) AWD-LSTM consists of three LSTMs with various regularizations. For the hyperparameters, we used the same values as \newciteDBLP:journals/corr/abs-1711-03953 except for the dropout rate for vector kj,ct and the non-monotone interval. We selected 0.6 because this value achieved the best score on the PTB validation dataset. For the non-monotone interval, we adopted the same value as \newcitefraternal.