Sequential recurrent neural networks for language modeling
Table 2: LMs performance on the PTB test set.
['N-1=', 'model 1', 'model 2', 'model 4', 'model+KN5 1', 'model+KN5 2', 'model+KN5 4', 'NoP 4', 'w/s 4']
We can clearly see that the proposed approach outperforms all other models using the lowest Number of model Parameters (NoP) among all configurations. We can also conclude that the explicit modeling of short range dependencies through sequential connections improves the performance. More precisely, the results show that increasing the history window (1, 2 and 4) improves the performance for all SRNN models. Furthermore, the word-dependent (WD-SRNN) model slightly outperforms the word-independent model (WI-SRNN) but with a non-negligible increase in the number of parameters. Regarding the training speed, we can conclude that training an SRNN model requires approximately twice the time needed for FFN and RNN, whereas it needs less time compared to LSTM.