TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
TABLE IV: We investigate how different amount of pre-training data affects performance by reporting ASR results on the LibriSpeech [librispeech] test-clean subset. For all the models, we limit the amount of labeled data to 100 hours to demonstrate the effect of pre-training.
['ModelsPre-train', '100 hr WER', '100 hr Rescore', '460 hr WER', '460 hr Rescore', '960 hr WER', '960 hr Rescore']
We further apply TERA to speech recognition tasks. Similar to the previous experiments, we show the results of TERA pre-trained with different combinations of objectives, and with different amount of pre-training data. Here we use the Kaldi [kaldi] fglarge lattice for LM rescoring. The LM rescored WER is denoted as Rescore. All ASR models are trained with 100 hours of labels from the train-clean-100 subset. TERA here is used for representation extraction (from the last layer), i.e., TERA parameters were frozen when adopting the DNN/HMM framework to speech recognition, and TERA representations are fed to the liGRU model as input. Applying all of the three objectives yields the best performance, whereas applying the channel objective on top of time is not always better than time objective only. This is because the channel objective mainly helps encode speaker information. However, utilizing all of the three auxiliary objectives can eliminate this disadvantage. Using time + channel + mag with 100 hours of pre-training data achieved similar performance (8.32% / 6.01%) to using time with 960 hours (8.31% / 5.99%). We see that 100 hours with three alteration objectives have a similar effect as 960 hours of data. This concretes the fact that using all three auxiliary objectives is potentially increasing the amount of pre-training data (or diversity of data) and eventually leads to improvement. As for the case of 960 hours of pre-training data, using different combinations of auxiliary objectives does not improve performance.