End-to-end Training Approaches for Discriminative Segmental Models
Table 7: Average cross entropy over frames before and after end-to-end fine-tuning.
['LSTM', 'train CE', 'dev CE', 'dev PER']
We have seen that end-to-end training initialized with a two-stage system leads to the best results. Since in end-to-end training, the meaning of the intermediate representations is not enforced anymore, it is unclear how the intermediate representations deviate from the learned ones. To answer this, we measure per-frame cross entropy for the LSTM frame classifier after end-to-end training. First, the per-frame cross entropy for the best performing LSTM on the training set can be as low as 0.06, which shows that a 3-layer bidirectional LSTM with 256 cells per layer is able to essentially memorize the entire TIMIT dataset. However, it is severely overfitting. Early stopping and dropout help balance cross entropies on the training set and development set. In addition, the cross entropies on both sets drop after end-to-end training. It shows that the meaning of the intermediate representations is still maintained by the LSTMs after end-to-end training.