Long-Short Range Context Neural Networks for Language Modeling
Table 3: LMs performance on the LTCB test set.
['Context Size M=N-1', 'model 1', 'model 2', 'model 4', 'NoP 4']
The LTCB experiments use the same PTB setup with minor modifications. More precisely, these results were obtained without use of momentum or weight decay (due to the long training time required for this corpus), the mini-batch size was set to 400, the learning rate was set to 0.4 and the BPTT step was fixed at 5. The FFNN and FOFE architectures use 2 hidden layers of size 600, whereas RNN, LSTM and LSRC have a single hidden layer of size 600. Moreover, the word embedding size was set to 200 for all models except RNN, which was set to 600. We also report results for an LSTM with 2 recurrent layers as well as for LSRC with an additional non-recurrent layer. In particular, we can see that the proposed LSRC model largely outperforms all other models. In particular, LSRC clearly outperforms LSTM with a negligible increase in the number of parameters (resulting from the additional 200×200=0.04M local connection weights Ucl) for the single layer results. We can also see that this improvement is maintained for deep models (2 hidden layers), where the LSRC model achieves a slightly better performance while reducing the number of parameters by ≈2.5M and speeding up the training time by ≈20% compared to deep LSTM.