Reducing Bias in Production Speech Models
Table 4: A deeper look into the dev set, measuring WER with language model decoding of each model on different slices of the dev set. Serving latency (milliseconds) is the 98th percentile latency on the last packet as described in Sec. 4. Training time is in hours per epoch with the data and infrastructure the same. *This model has twice the number of parameters as the Baseline, and suffers from prohibitively large serving latency.
['Devsets', 'Baseline', '2×Baseline*', 'Mix-3']
The model mix of joint training with 3 increasingly difficult losses (CE, CTC, and GramCTC, Mix-1) achieves the best results on the train set far surpassing the other model mixes, and even nearly matching the performance of models with bidirectional layers on the train set. This model has the smallest gain on the dev set amongst all the mix-models, and puts it in the overfitting regime. We know that there exists a model that can generalize better than this one, while achieving the same error rates on the train set: the bidirectional baseline. The model mix of PCEN, LC-BGRU and IR augmentation (Mix-2) performs worse on the train set – additional data augmentation with IR impulses makes the training data harder to fit as we have seen earlier, but PCEN and LC-BGRU is not sufficient to address this difficulty. However, the model does attain better generalization and performs better on the dev set, and actually surpasses our bidirectional target when using a language model. Mix-3 adds CE joint training which helps to address optimization issues and leads to lower error rates on both the train and dev sets. However, the improvement in dev WER disappears when using a language model, again highlighting the language model integration issues when using CE training. Clearly, Mix-3 is significantly better for “farfield” and “Names” speech data, two notably difficult categories for ASR. ASR tasks run into a generalization issue for “Names” categories because they are often required words that is not present in the acoustic training data. Similarly far field audio is hard to obtain and the models are forced to generalize out of the training data, in this case by making use of augmentation. At the same time, the serving latency of Mix-3 is only slightly higher than baseline model, still good for deployment.