Persistent Hidden States and Nonlinear Transformation for Long Short-Term Memory
Table 2: Negative log-likelihood of language models for the Wikitext-103 dataset
['Models', 'Sentence Length ≤ 50 Valid NLL', 'Sentence Length ≤ 50 Test NLL', 'Length ≤ 250 Valid NLL', 'Length ≤ 250 Test NLL']
To see how the proposed methods work with large-scale data, we added language modeling task with Wikipedia Text data where the training data size is 516MB. Table. From the table, we can see that when the length is short (less than or equal to 50), the RNN units have the same (or very similar) performance. However, when the length is large, the proposed methods outperform the conventional LSTM. It indicates that the proposed units can keep past information better than LSTM. Also, the nonlinear transformation is not crucial for word-level language model.