What does a network layer hear? Analyzing hidden representations of End-to-end ASR through speech synthesis
Table 2: EER results obtained by the ThinResNet[19] on the reconstructed speech utterances.
['[EMPTY]', 'SNR', 'type', 'VGG-LSTM cnn1', 'VGG-LSTM cnn2', 'VGG-LSTM cnn4', 'VGG-LSTM blstm1', 'VGG-LSTM blstm3', 'VGG-LSTM blstm5', 'pure-LSTM blstm1', 'pure-LSTM blstm2', 'pure-LSTM blstm3', 'pure-LSTM blstm4', 'pure-LSTM blstm5']
We reconstruct a bunch of speech utterances from every layer of the 4 kinds of ASR models. Then we randomly sample the same number of positive and negative pairs to test the above two speaker verification models to estimate the amount of speaker information. (We only present the results from ThinResNet because the results from two speaker verification models are quite similar.) The two big columns represent the ASR network architectures, and from left to right is from the top layer to the deepest layer. Rows(a, c, e, g) mean that we take the speech at different SNR ratio as the robust VGG-LSTM or pure-LSTM inputs, while rows(b, d, f, h) are to feed the baseline ASR models. If we listen to the recovered speech, we can directly recognize that utterances of different speakers from the deeper layer become more indistinguishable from each other than shallower layer. Meanwhile, the EER values are increased along the layers of all models, meaning that the speaker characteristics are gradually removed out steps by steps.