TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
TABLE I: Frame-wise phone classification results on LibriSpeech. The training and testing sets are all identical to the ones used in the CPC [cpc] literature. We present testing set accuracy (%) from Linear classifiers and 1 Hidden layer classifiers, measured across various auxiliary objectives and amount of pre-training data.
['RepresentationPre-train', '100 hr Linear', '100 hr 1 Hidden', '360 hr Linear', '360 hr 1 Hidden', '960 hr Linear', '960 hr 1 Hidden']
Comparison of different network depth and size. We show results of TERA pre-trained with different combinations of objectives, and with different amount of pre-training data. TERA here is used for representation extraction (from the last layer), i.e., TERA parameters were frozen when adapting classifiers to phoneme classification. The classification accuracy is compared with four different baselines: MFCC, FBANK, fMLLR features, and CPC representations [cpc]. For the Linear classifier, FBANK outperforms MFCC, and fMLLR outperforms FBANK in terms of linear separability. For the 1 Hidden classifier, fMLLR again achieves the highest performance among surface features. On the other hand, as expected TERA and CPC representations outperform all surface features for both types of classifiers.