Generative Pre-training for Speech with Autoregressive Predictive Coding
Table 2: ASR WER results with varying amounts of training data randomly sampled from si284. Feature extractors pre-trained with just 10 hours of LibriSpeech audio are denoted with a subscript 10.
['Features', 'Proportion of si284 1', 'Proportion of si284 1/2', 'Proportion of si284 1/4', 'Proportion of si284 1/8', 'Proportion of si284 1/16', 'Proportion of si284 1/32']
For example, 1/16 means that we take only 72×1/16=4.5 hours from si284 for training. We find that for all input features, there is a significant increase in WER whenever the training size is reduced by half. When comparing R-APC and T-APC with log Mel, we see the former two always outperform the latter across all proportions, and the gap becomes larger as training size decreases. Note that when using only half of si284 for training, R-APC already matches the performance of log Mel trained on the full set (18.3), and T-APC even outperforms it (16.4 vs. 18.3). In particular, we observe that T-APC always outperforms log Mel by using half of the training data log Mel uses. This observation aligns with the findings in recent NLP literature Finally, we see that most of the time APC outperforms CPC and PASE. In some cases PASE is slightly better than T-APC10 (e.g., when only 1/8 or less of si284 is available), but is still worse than R-APC10.