Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Table 5: Ablation study on Penn Treebank and WikiText-2 without finetuning or dynamical evaluation.
['[BOLD] Model', 'PTB  [BOLD] Validation', 'PTB  [BOLD] Test', 'WT2  [BOLD] Validation', 'WT2  [BOLD] Test']
Compared to the vanilla AWD-LSTM, though being more expressive, MoC performs only better on PTB, but worse on WT2. It suggests that simply adding another hidden layer or employing a mixture structure in the feature space does not guarantee a better performance. On the other hand, training AWD-LSTM using MoS hyper-parameters severely hurts the performance, which rules out hyper-parameters as the main source of improvement.