Why Self-Attention?A Targeted Evaluation of Neural Machine Translation Architectures
Table 4: The results of different models with small training data and replicate settings. Trans is short for Transformer. Models with the suffix “-s” are models trained with small data set. Models with the suffix “-re” are models trained with replicate settings. “h2, h4, h8” indicates the number of attention heads for Transformer models.
['Model', '2014', '2017', 'PPL', 'Acc(%)']
A first hypothesis is that this is caused by the amount of training data, since we used much larger datasets than \newciteTran2018recurrent. We retrain all the models with a small amount of training data similar to the amount used by \newciteTran2018recurrent, about 135K sentence pairs. We do not see the expected degradation of Transformer-s, compared to RNNS2S-s