A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
Table 1: Details of pre-training corpora
['Datasets', 'Hours', 'Speaking Style']
For Mandarin pre-training, we first collected reading type dataset OpenMandarin as described in [Jiang2019ImprovingTS]. Note unlike [Jiang2019ImprovingTS], we did not include HKUST and AISHELL in OpenMandarin dataset in this work because we found self pre-trained MPC also improves performance. OpenMandarin contains about 1150 hours of speech in total. To understand the impact of pre-training data speaking style, our internal reading type dataset Didi Dictation and spontaneous type dataset Didi Callcenter were also included. Didi Dictation is collected from our internal mobile dictation application while Didi Callcenter is collected from phone calls between our user and customer service staff. We randomly selected 5000 hours of Didi Callcenter and 5000 hours of Didi Dictation for MPC pre-training. For fair comparison with Open Mandarin, we also added another pre-training dataset by randomly selecting 1150 hours of Didi Callcenter and name it Didi Callcenter - 1K. For English pre-training data, reading type dataset Librispeech [Panayotov2015LibrispeechAA] and spontaneous type dataset Fisher [Cieri2004TheFC] are used. To provide a fair comparison on data size, we created a new pre-training dataset by randomly selecting only 960 hours of Fisher dataset and name it Fisher - 1K.