Effectiveness of self-supervised pre-training for speech recognition
Table 3: WER on Librispeech with no pre-training for continuous and discrete input features on 100 hours of labeled data.
['Input features', 'dev clean', 'dev other', 'test clean', 'test other']
To better understand the impact of BERT pre-training in our representation learning approach, we remove the BERT pre-training step and only perform unit discovery through vq-wav2vec, for discrete inputs, and fine-tuning, for both discrete and continuous inputs on the 10 hour labeled setup. The vocabulary for discrete inputs is still built on the unlabeled data. This is likely because the representations of the input discrete units are random and training on the labeled data is not sufficient. Continuous inputs do not suffer from this issue.