Effectiveness of self-supervised pre-training for speech recognition
Table 1: Librispeech WER of BERT models taking discretized or continuous input features: for discretized BERT quantized input units are obtained via vq-wav2vec or by k-means clustering of MFCC/FBANK features; for continuous BERT wav2vec features are learned following [12]. Pre-trained models are fine-tuned on various sizes of labeled data.
['Model', 'Input features', 'dev clean', 'dev other', 'test clean', 'test other']
The first observation is that Discrete BERT outperforms Continuous BERT in all settings. This shows that pre-training over meaningful discrete units outperforms directly learning representations from the continuous unlabeled data. The initial unit discovery builds a vocabulary that makes the subsequent BERT pre-training more effective.