Accurate Word Alignment Induction from Neural Machine Translation
Table 6: Comparison of Transformer-small and Transformer-big. Note that Transformer-small is trained on the datatset discussed in Section 5.1, while Transformer-big is provided by fairseq-py and trained on WMT19 for de→ en translation, WMT16 for en→ de translation and WMT14 for en→ fr translation.666We do not compare on other language pairs because fairseq does not provide translation models on those translation tasks.
['Transformer-small', 'BLEU', 'de→en 33.20', 'en→de 25.80', 'de-en bidir -', 'en→fr 34.84']
In this experiment, we test whether good word alignment can be extracted from the Transformer model under other configurations. Instead of training the model from scratch by ourselves, we utilize the pretrained Transformer models provided by fairseq-py. All the models are based on the configuration of transformer_wmt_en_de_big, therefore we denote them as Transformer-big. It shows that although the BLEU score obtained by Transformer-small and Transformer-big can be different, the AER scores of the two models with align-att are similar. We also list the layer from which align-att extracts alignment. Transformer-small and Transformer-big both have 6 decoder layers. However, the best layer for Transformer-small is always layer 3, while for Transformer-big, the best layer is layer 2 for de→en and en →de translations. This indicates that with more data and big model, the Transformer tends to acquire alignment information at lower layer.