A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
Table 2: Character Error Rates(%) and Relative Error Rates Reduction(%) on HKUST and AISHELL test set
['Task', 'Pre-training Data', 'Hours', 'CER', 'RERR']
Our baseline result without MPC matches the strong baseline in [karita]. Comparing relative error reduction of HKUST with same amounts of pre-training data, it is obvious MPC models pre-trained with matching speaking style data achieved lower error rates for downstream tasks. Open Mandarin and Didi Dictation are both used in pre-training for HKUST and AISHELL. Interestingly, the error reduction Didi Callcenter - 1K brings on HKUST is even bigger than Didi Dictation, which suggests speaking style has a bigger impact than pre-training data size in some cases.