A Study on Neural Network Language Modeling
Table 5: Evaluating NNLM on data sets from different fields
['Model', '[ITALIC] m', '[ITALIC] nh', 'Training Data Electronics', 'Training Data Books', 'PPL Electronics', 'PPL Books']
Since the training of neural network language model is really expensive, it is important for a well-trained neural network language model to keep learning during test or be improved on other training data set separately. However, the neural network language models built so far do not show this capacity. ANN is just a numerical approximation method in nature, and it approximate the target function, the probabilistic distribution of word sequences for LM, by tuning parameters when trained on data set. The learned knowledge is saved as weight matrixes and vectors. When a trained neural network language model is expected to adaptive to new data set, it should be retrained on both previous training data set and new one. This is another limit of NNLM because of knowledge representation, i.e., neural network language models cannot learn dynamically from new data set.