TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
TABLE VIII: Comparison of pre-training approaches between recent work and the proposed approach on TIMIT [timit]. All the pre-training data are from LibriSpeech [librispeech], if not specified otherwise. All of the TERA models use the combined auxiliary objective of time + channel + mag alteration.
['Models', 'Pre-train', 'PER']
We then explore how the mismatch of domains between pre-training and downstream tasks affects performance. For the exploration, we pre-train TERA with LibriSpeech [librispeech], and apply the resulting networks to the supervised TIMIT [timit] ASR task. The same Hybrid ASR setting and framework as described above for LibriSpeech ASR is used, except that we adopt a learning rate of 4e^{-4} and a batch size of 8. We also list the results of strong supervised systems [tdfilter, cnnphone, ligru]. All of the TERA models use a combination of time + channel + mag alteration as the auxiliary objectives, and are pre-trained with various amount of data. As expected, pre-training on larger amount of data gives performance benefit, and we achieved the best WER (14.5%) with 960 hours of pre-training data. We find that for TIMIT ASR as the downstream task, using either WS or FT is not helpful, and extracting speech representations from the last layer provides the best performance. Also, there is no significant gain when extracting features from a larger model medium. The reason is likely because there is not enough labeled data in TIMIT. On the other hand, our best model (14.5%) outperforms wav2vec [wav2vec] (14.7%) that uses both LibriSpeech [librispeech] and Wall Street Journal (WSJ) data for pre-training, as well as all the strong supervised baselines. By using only 100 hours of pre-training data, we even surpass the result of wav2vec that uses 960 hours for pre-training (15.2% v.s. 15.6%).