A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation
Table 1: Hyper-parameters used to train Recurrent and Transformer models, unless differently specified.
['[EMPTY]', 'encoder-decoder', 'embedding', 'hidden', 'encoder', 'decoder', 'batch']
Each of the three system types, namely bilingual, multilingual and zero-shot, is trained using both Recurrent and Transformer architectures, with the proper training data provided in the IWSLT 2017 evaluation campaign. Meta training parameters were set in a preliminary stage with the aim of maximizing the quality of each approach. Hence, we took the precaution of selecting the optimal training and inference parameters for both approaches and toolkits. (RNN) or 0.2 (Transformer) is used. If the perplexity does not decrease on the validation set or the number of epochs is above 7, a learning rate decay of 0.7 is applied in the Recurrent case.