A Study on Neural Network Language Modeling
Table 6: Examine the learning ability of neural network
['Model', '[ITALIC] m', '[ITALIC] nh', 'Dynamic', 'PPL']
The lowest perplexity on each test data set was gained by the model trained on corresponding training data set, instead of the model trained on both training data set The results show that the knowledge represented by a neural network language model is the probabilistic distribution of word sequences from training data set which varies from field to field. Except for the probabilistic distribution of word sequences, the feature vectors of words in vocabulary are also formed by neural network during training. Because of the classification function of neural network, the similarities between words can be observed using these feature vectors. However, the similarities between words are evaluated in a multiple dimensional space by feature vectors and it is hard to know which features of words are taken into account when these vectors are formed, which means words cannot be grouped according to any single feature by the feature vectors. In summary, the knowledge represented by neural network language model is the probabilistic distribution of word sequences from certain training data set and feature vectors for words in vocabulary formed in multiple dimensional space. Neither the knowledge of language itself, like grammar, nor the knowledge conveyed by a language can be gained from neural network language models. Therefore, NNLM can be a good choice for NLP tasks in some special fields where language understanding is not necessary. Language understanding cannot be achieved just with the probabilistic distribution of word sequences in a natural language, and new kind of knowledge representation should be raised for language understanding.