Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation
Table 2: Speech recognition results on HKUST Chinese dataset. “Valid” is the character error rate (CER) on the valid set and “Test” is the CER on the test set. “+” denotes the log-linear interpolation with equal weights. Other columns has the same meanings as in Table 1.
['Model', 'Valid (%)', 'Test (%)', '#param (M)', 'Training time', 'Inference time']
First, our neural TRF LM outperforms the classical 5-gram LM (denoted by “KN5”) with relative CER reduction 2.7%, and performs close to “LSTM-2×200” (a LSTM LM with 2 hidden layers, 200 hidden units per layer and a standard softmax output layer) with 63% parameters. Second, even though the vocabulary in this experiment is small, containing only about 4000 characters, our neural TRF LM is still 5 times faster than LSMT LM in rescoring sentences. Third, the lowest WER 26.87% is achieved by combining the neural TRF LM with the 5-gram LM and the LSTM LM (denoted by “KN5+LSTM+neural TRF”). Finally, together with the previous experiment on PTB dataset, these results demonstrate the language independence in applying neural TRF LMs.