Accurate Word Alignment Induction from Neural Machine Translation
Table 1: AER on the test set with different alignment methods. bidir are symmetrized alignment results. Since we use the same dataset and preprocessing as Ding et al. (2019), for Attention, SmoothGrad, SD-SmoothGrad and AddSGD, the results are quoted from Ding et al. (2019). Since Garg et al. (2019) report that BPE is beneficial for statistical alignment models, which is consistent with our own experiments, we report the BPE-based fast-align and GIZA++ results. The column Modify represents whether the method modifies the training loss (loss) or the Transformer architecture (arch). The column Fullc denotes whether full-context is used to induce alignment at test time. Best bidir neural results that induce alignment without modification of loss or architecture are marked with underlines, and comparable best bidir results among all methods are marked with boldface.
['Method', 'Modify', 'Fullc', 'de-en de→en', 'de-en en→de', 'de-en bidir', 'fr-en fr→en', 'fr-en en→fr', 'fr-en bidir', 'ro-en ro→en', 'ro-en en→ro', 'ro-en bidir']
It shows that our method align-att significantly outperforms all neural baselines that extract alignment from vanilla Transformer (Modify: -). For example, it outperforms SD-SmoothGradLi et al. Our method align-att also outperforms align-loo in terms of both alignment quality and induction computation cost. The success of align-att shows that vanilla Transformer has captured alignment information in an implicit way, which could be revealed if the correct decoding step and layer are chosen to induce alignment. we also compare our methods with statistical alignment methods. align-att significantly outperforms fast-align, both under online and offline settings. It achieves slightly worse performance compared with GIZA++. Since align-att is better than align-loo across all datasets, we focus on experiments with align-att in the remaining sections.