Language modeling with Neural trans-dimensional random fields
Table 2: Performances of various LMs. “PPL” is the perplexity on PTB test set. “WER” is the word error rate on WSJ’92 test data. “#param” is the number of parameter numbers (in millions). “+” denotes the log-linear interpolation with equal weights of 0.5. For LSTMs, the WER is obtained using the reset strategy. “Inference time” denotes the average time of rescoring the 1000-best list for each utterance.
['Model', 'PPL', 'WER(%)', '#param (M)', 'Training time', 'Inference time']
Compared with “LSTM-2x200” with similar model size, the neural TRF LM achieves a relative WER reduction of 4.5%. Compared with “LSTM-2x650”, the neural TRF LM achieves a slightly lower WER with only a fifth of parameters. The large “LSTM-2x1500” performs slightly better than the neural TRF but with 16.5 times more parameters. Third, we examine how neural TRF LMs and LSMT LMs are complimentary to each other. The probability of each sentences are log-linearly combined with equal interpolated weights of 0.5. The interpolated neural TRF and “LSTM-2x1500” further reduces the WER and achieves the lowest WER of 7.17.