Automatic Speech Recognition with Very Large Conversational Finnish and Estonian Vocabularies
TABLE III: Comparison of uniform data processing, random sampling of web data by 20 %, and weighted parameter updates from web data by a factor of 0.4, in NNLM training. The models were trained using normal softmax. Includes development set perplexity, word error rate (%), and word error rate after interpolation with the n-gram model.
['Subset Processing', 'Training Time', 'Perplexity', 'WER', '+NGram']
Optimizing the weights for neural network training is more difficult than for the n-gram mixture models. As we do not have a computational method for optimizing the weights, we tried a few values, observing the development set perplexity during training. Sampling 20 % of the web data on each iteration, or weighting the web data by a factor of 0.4 seemed to work reasonably well. We used a slightly higher learning rate when weighting the web data to compensate for the fact that the updates are smaller on average. Uniform means that the web data is processed just like other data sets , sampling means that a subset of web data is randomly sampled before each epoch, and weighting means that the parameter updates are given a smaller weight when the mini-batch contains web sentences. Sampling seems to improve perplexity, but not word error rate. Because sampling usually speeds up training considerably and our computational resources were limited, the rest of the experiments were done using sampling.