Deep LSTM for Large Vocabulary Continuous Speech Recognition
Table 4: The CER of different 2-layers models, which are Shenma distilled model, Amap model further trained with Amap dataset, and Shenma model trained with sMBR on Amap dataset.
['Training methods', 'CER (%)']
2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model. It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search. Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training. If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough. Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR. 2-layers Shenma model further trained with sMBR achieves about 8.1% relative reduction, compared with 2-layers regular-trained Amap model. Both training sMBR datasets contain the same 1000 hours data. As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data. Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time.