UNSUPERVISED PRE-TRAINING FOR SEQUENCE TO SEQUENCE SPEECH RECOGNITION
Table 2: CER[%] performance of our unsupervised pre-training method on AISHELL-1.
['Exp', 'Models', '10h', '20h', '89h', '178h']
In this section, we evaluate our approach on AISHELL-1, whose distribution is consistent with AISHELL-2. We use 20 hours, 40 hours, 89 hours and 178 hours of AISHELL-1 as in-domain paired data for post-training respectively. A0 is the baseline model initialized with scratch. Initialized with our unsupervised pre-trained model (M2), A1 obtains a relative CERR from 38.24% to 7.88% than baseline system (A0) on the four ratios of AISHELL-1, which indicates that our unsupervised pre-training benefits downstream ASR tasks. And it is obvious that the smaller the amount of in-domain data is, the more significant the improvement is. Besides, models initialized with our unsupervised pre-training method converge consistently faster than randomly initialized baseline. Fig. In another two case, the loss curves are similar. Expertiments A4 are initialized with a supervised pre-training model M3. If baseline system (A0) is a lower bounds of our unsupervised pre-training, A4 can be regarded as upper bounds. From A1 and A4, our unsupervised pre-training has been pretty close to the supervised pre-training. In this section, we use several ablation studies to demonstrate the effect of acoustic and linguistic pre-training respectively. A2 obtains relative CERR from 23.10% to 5.34% with only acoustic pre-training and A3 obtains relavtive CERR from 28.87% to 3.94% with only linguistic pre-training. It means that two stages of unsupervised pre-training can boost the downstream ASR task independently. And when the amount of available paired data for ASR task is very small (10 hours), the linguistic pre-training is more useful than acoustic pre-training. With the amount of paired data increasing, the acoustic pre-training starts to play a more important role.