RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
Table 1: Training speed and memory consumption on WMT 2017 German→English. Train time is for seeing the full train dataset once. Batch size is in words, such that it almost maximizes the GPU memory consumption. The BLEU score is for the converged models, reported for newstest2015 (dev) and newstest2017. The encoder has one bidirectional LSTM layer and either 3 or 5 unidirectional LSTM layers.
['toolkit', 'encoder n. layers', 'time [h]', 'batch size', 'BLEU [%] 2015', 'BLEU [%] 2017']
We want to compare different toolkits in training and decoding for a recurrent attention model in terms of speed on a GPU. Here, we try to maximize the batch size such that it still fits into the GPU memory of our reference GPU card, the Nvidia GTX 1080 Ti with 11 GB of memory. We keep the maximum sequence length in a batch the same, which is 60 words. We always use Adam Kingma and Ba For these speed experiments, we did not tune any of the hyper parameters of RETURNN which explains its worse performance. The aim here is to match Sockeye’s exact architecture for speed and memory comparison. During training, we observed that the learning rate scheduling settings of Sockeye are more pessimistic, i.e. the decrease is slower and it sees the data more often until convergence. This greatly increases the total training time but in our experience also improves the model.