Feature Learning in Deep Neural Networks – Studies on Speech Recognition Tasks
Table 3: Euclidean distance for the output vectors at each hidden layer (L1-L7) and the KL divergence (nats) for the posteriors at the top layer between the narrowband (8 kHz) and wideband (16 kHz) input features, measured using the wideband DNN or the mixed-bandwidth DNN.
['layer', 'dist', 'wideband DNN mean', 'wideband DNN variance', 'mixed-band DNN mean', 'mixed-band DNN variance']
To understand the difference between these two scenarios, we take the output vectors at each layer for the wideband and narrowband input feature pairs, hℓ(xwb) and hℓ(xnb), and measure their Euclidean distance. For the top layer, whose output is the senone posterior probability, we calculate the KL-divergence in nats between py|x(sj|xwb) and py|x(sj|xnb). Perhaps what is more interesting is that the average distances and variances in the data-mixed DNN are consistently smaller than those in the DNN trained on wideband speech only. This indicates that by using mixed-bandwidth training data, the DNN learns to consider the differences in the wideband and narrowband input features as irrelevant variations. These variations are suppressed after many layers of nonlinear transformation. The final representation is thus more invariant to this variation and yet still has the ability to distinguish between different class labels. This behavior is even more obvious at the output layer since the KL-divergence between the paired outputs is only 0.22 in the mixed-bandwidth DNN, much smaller than the 2.03 observed in the wideband DNN.