TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
TABLE II: Comparison of different network depth and size. LibriSpeech [librispeech] frame-wise phone classification results are presented, with the same phone set, train and test set as in Table I.
['Representation', '#L', '#ParamPre-train', '100 hr Linear', '100 hr 1 Hidden', '360 hr Linear', '360 hr Concat', '960 hr Linear', '960 hr 1 Hidden']
Furthermore, we investigate how the network depth (number of layers) affects the performance of TERA in phoneme classification. Classification accuracy of four TERA models: base, medium, large, and xlarge, is presented together with CPC [cpc] and Modified CPC [modified_cpc]. The scores of CPC and Modified CPC are from the original literature. All of the TERA models use the full objective of time + channel + mag. For the case where models are pre-trained with 100 hours of data, all the TERA models outperformed CPC. When 360 hours of data is used for pre-training, we report Concat instead of 1 Hidden to compare TERA with Modified CPC. With 360 hours of data, the base (66.4% / 68.3%) and medium (66.6% / 68.9%) achieved comparable result with Modified CPC (- / 68.9%) that has a network depth of 6 layers, and as the size of TERA increases, TERA representations achieve better separability and yield better performance than Modified CPC. In the case where 100/360 hours of pre-training data is used, a larger model tends to show performance benefit. As for the case where 960 hours of pre-training data is used, model size tends to be irrelevant and all the TERA models achieved similar performance. On the other hand, we also observe a correlation between performance and data size. As we increase the amount of pre-training data, TERA usually yields better performance.