Transferring Knowledge from a RNN to a DNN
Table 3: Cross Entropy Error (CSE) on WSJ dev93 over our various models.
['[BOLD] Alignment', 'Model', 'CSE']
We compute the CSE of our various models against the GMM alignment for the dev93 dataset. We measure the CSE against dev93 since that is our stopping criteria and that is the optimization loss. The CSE will give us a better indication of the optimization procedure, and how our models are overfitting. There are several observations, first the big RNN is able to achieve a lower CSE compared to the big DNN. The RNN model is able to optimize better than the DNN as seen with the better WERs the RNN model provides. This is as expected since the big RNN model achieves the best WER.