Sequential recurrent neural networks for language modeling
Table 3: LMs Perplexity on the LTCB test set.
['Context Size M=N-1', 'model 1', 'model 2', 'model 4', 'NoP 4']
The LTCB experiments use the same PTB setup with minor changes. More precisely, these results were obtained without usage of momentum or weight decay whereas the mini-batch size was set to 400. The FNN and FOFE architectures contain 2 hidden layers of size 600 (or 400) whereas RNN and SRNN have a single hidden layer of size 600. In particular, we can see that SRNN models outperform all other models while requiring comparable or fewer model parameters. Moreover, the WI-SRNN∗ model with a single hidden layer slightly outperforms FOFE (2 hidden layers). These results, however, show a more significant improvement for the WD-SRNN model and for the increased window size (from 1 to 4) compared to the improvement obtained on the PTB. This is mainly due to the large amount of LTCB training data, which allows us to train richer WD context vectors.