Generative Pre-training for Speech with Autoregressive Predictive Coding
Table 3: ASR WER results using different numbers of GRU layers for the encoder in the ASR seq2seq model.
['Features', 'Number of encoder layers 1', 'Number of encoder layers 2', 'Number of encoder layers 3', 'Number of encoder layers 4']
The next aspect we examine is to what extent can we reduce the downstream model size with transfer learning. We see that when using the same number of layers, * -APC and *-APC10 always outperform other features. It is noteworthy that T-APC with just 2 layers performs similar to log Mel using 4 layers (18.6 vs. 18.3), which demonstrates the effectiveness of APC transfer learning for reducing downstream model size.