Why Self-Attention?A Targeted Evaluation of Neural Machine Translation Architectures
Table 7: The performance of CNN models trained by different toolkits. K is the kernel size of CNN.
['Model', 'Layer', 'K', '[ITALIC] 2014', 'Accuracy(%)']
The pre-trained Fairseq model has 15 layers, which is much deeper than the Sockeye models that we trained. It achieves higher BLEU score on newstest2014, and higher accuracy on modeling long-range dependencies, than the 8-layer Sockeye models that we trained. However, it still lags behind RNNS2S and Transformer on the subject-verb agreement task.