Persistent Hidden States and Nonlinear Transformation for Long Short-Term Memory
Table 1: Negative log-likelihood of language models for Penn TreeBank data
['Models', 'Validation NLL (V)', 'Test NLL (T)', 'Generalization (V-T)']
Table. Since nonlinear transformation might be crucial to a language model, we included LSTM+ which is implemented in the same way as PRU+, except that it still has the term \boldmathU\boldmathht−1 for the cell state update. From the table, we can see LSTM+ is not helping which might imply that LSTM already has enough power for nonlinear transformation. However, PRU has better performance compared to LSTM, and PRU+ is almost the same as (if not slightly better than) PRU. It may mean that PRU is successful in memorizing information and transforming it, and PRU+ can add slightly more nonlinear transformation effect without losing memorization efficiency. In other words, PRU is so simple that it can effectively memorize past information, so it improves the performance by only memorizing well. Then, PRU+ improves the performance by adding nonlinear transformation power on top of the advantage of PRU.