UNSUPERVISED PRE-TRAINING FOR SEQUENCE TO SEQUENCE SPEECH RECOGNITION
Table 3: CER[%] performance of our unsupervised pre-training method on HKUST.
['Exp', 'Models', '1/4', '1/2', '3/4', 'ALL']
The distribution of AISHELL-1 and AISHELL-2 is similar. But in most case, the distribution of paired data used in post-training is very different from the data used in pre-training procedure. In this section, we evaluate our approach on one fourth, half, three fourths and all of HKUST dataset respectively, which is very different from AISHELL-2 used in pre-training. The results of B3 is much worse than B1, and it indicates that in such a case (there is distribution mismatch between pre-training data and in-domain data), acoustic pre-training are more useful than linguistic pre-training. Comparing B2 with B1, we find that linguistic pre-training may bring negative effects when we have applyed acoustic pre-training in such a mismatch case.