RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
Table 6: Pretraining comparison.
['encoder num. layers', 'BLEU [%] no pretrain', 'BLEU [%] with pretrain']
RETURNN supports very generic and flexible pretraining which iteratively starts with a small model and adds new layers in the process. A similar pretraining scheme for deep bidirectional LSTMs acoustic speech models was presented earlier Zeyer et al. Here, we only study a layer-wise construction of the deep bidirectional LSTM encoder network of an encoder-decoder-attention model for translation on the WMT 2017 German→English task. The observations very clearly match our expectations, that we can both greatly improve the overall performance, and we are able to train deeper models. A minor benefit is faster training speed of the initial pretrain epochs.