Direct Output Connection for a High-Rank Language Model
Table 4: Coefficient of variation of Equation 10: √β in validation and test sets of PTB.
['[ITALIC] λβ', 'Valid', 'Test']
For the i3=15,i2=5 setting, we explored the effect of λβ in {0,0.01,0.001,0.0001}. This table demonstrates that the coefficient of variation decreases with growth in λβ. In other words, the model trained with a large λβ assigns balanced weights to each probability distribution. These results indicate that it is not always necessary to equally use each probability distribution, but we can acquire a better model in some λβ. Hereafter, we refer to the setting that achieved the best score (i3=15,i2=5,λβ=0.001) as AWD-LSTM-DOC.