TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
TABLE III: Speaker linear classification results on LibriSpeech. We use identical training and testing sets as in the CPC [cpc] and AALBERT [audioalbert] literature. We show both frame-wise linear classification accuracy (Frame, %), and utterance-wise linear classification accuracy (Utterance, %) where we average representations over time.
['RepresentationPre-train', '100 hr Frame', '100 hr Utterance', '360 hr Frame', '360 hr Utterance', '960 hr Frame', '960 hr Utterance']
We show results of TERA pre-trained with different combinations of objectives, and with different amount of pre-training data. Here TERA is used as a representation extractor (from the last layer) where parameters are frozen during adapting to speaker classification tasks. We compare the classification accuracy with five baselines: MFCC features, fMLLR features, CPC representations [cpc], AALBERT representations [audioalbert], and Mockingjay representations [audioalbert, mockingjay]. MFCC features perform poorly for the task of Frame since the MFCC extraction process eliminates most of the pitch information from speech, and only reserves partial tone characteristics. FBANK features perform even worse than MFCC for Frame. It shows that the speaker information encoded in FBANK is not linearly accessible. The fMLLR feature extraction process eliminates nearly all of the speaker information and yields an accuracy of 0.4% for Frame. The accuracy is similar to a random guess among the 251 speakers (1/251\approx 0.003984). On the other hand, when evaluating fMLLR and FBANK with Utterance, a slight improvement is observed, but the performances are still terrible, while MFCC does not improve for Utterance.