A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
Table 5: MPC + APC is the model pre-trained with APC 50% of time and MPC 50% of time. Relative Error Rates Reduction(%) is calculated with HKUST baseline without MPC
['Pre-training Data', 'Objective', 'CER', 'RERR']
To further improve performance of MPC on streaming models, we tried to combine APC with MPC in pre-training stage. Previous work on APC showed a future prediction step of 5 gave best results on Transformer decoder [Chung2019GenerativePF]. We implemented APC with the same future prediction step and set switching probability p to 0.5. When used alone, APC and MPC got similar improvements. Combining APC with MPC results in consistent gains over them, which echoes with the findings in [Dong2019UnifiedLM].