Generative Pre-training for Speech with Autoregressive Predictive Coding
Table 1: ASR results (WER ↓) of APC with varying n during pre-training and different transfer learning approaches (Frozen vs. Finetuned). log Mel is the baseline that uses log Mel spectrograms as input features. The best transfer learning result is marked in bold.
['Features', '[ITALIC] n 1', '[ITALIC] n 2', '[ITALIC] n 3', '[ITALIC] n 5', '[ITALIC] n 10', '[ITALIC] n 20']
We also include the case where APC is randomly initialized and trained from scratch along with the seq2seq model. We think this is because for a small n, APC can exploit local smoothness in the spectrograms for predicting the target future frame (since xk can be very similar to xk+n when n is small) and thus does not need to learn to encode information useful for inferring more global structures; an overly large n, on the other hand, makes the prediction task too challenging such that APC is unable to generalize across the training set. The best n for R-APC is 3 and for T-APC it is 5. We also find that for all n, keeping pre-trained APC weights fixed (*-APC Frozen), surprisingly, works better than fine-tuning them (*-APC Finetuned), while the latter still outperforms the baseline. Furthermore, we see that training APC from scratch along with the seq2seq model (*-APC Scratch) always performs the worst—even worse than the baseline. With APC transfer learning, WER is reduced by more than 25% from 18.3 to 13.7.