Language modeling with Neural trans-dimensional random fields
Table 1: Comparison of three different rescoring strategies for LSTM LMs. “LSTM-2×200/650/1500” indicates the LSTM with 2 hidden layers and 200/650/1500 hidden units per layer. “PPL” denotes the perplexity on the PTB test set. “WER-p”, “WER-r” and “WER-s” correspond to the preserve, reset and shuffle strategy respectively.
['LSTMs', 'PPL', 'WER-p', 'WER-r', 'WER-s']
It can be seen that the lowest WER is achieved by “LSTM-2×1500” with 2 hidden layer and 1500 hidden units, using the preserve strategy. When we shuffle the sentences, the WER increases significantly, from 7.32 to 8.18, which is even worse than the WER of “LSTM-2×200”. This is because that in the preserve strategy, the candidate sentences of each utterance are rescored successively. The final hidden state carries relevant information to benefit the prediction of the next candidate sentence which belongs to the same testing utterance as the current candidate sentence. After shuffling, the relation between adjacent candidate sentences is broken. The information preserved in the hidden states may mislead the prediction.