A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
Table 6: Results on HKUST and AISHELL with different knowledge transfer methods. HKUST is pre-trained with Didi Callcenter. AISHELL is pre-trained with Didi Dictation. Target Data + Layer-wise is the combination of target data adaption and layer-wise discriminative training
['Task', 'Knowledge Transfer Methods', 'CER', 'RERR']
Two baseline results we used are HKUST pre-trained with Didi Callcenter and AISHELL pre-trained with Open Mandarin. Using target data adaption or layer-wise discriminative training alone doesn’t help the knowledge transfer of MPC very much. But when combined together, they provide consistent gains on downstream tasks. For multi-task with MPC, the MPC loss should contribute more to the joint loss to facilitate knowledge transfer in the first few epochs. As training proceeds, the task-specific component of the loss function becomes more important and γmpc should be decreased. In this work, we empirically found it work well to set initial γmpc to 0.2 and decrease it by half every 5 epochs.