Evaluating Explanation Methods for Neural Machine Translation
Table 7: Relation with word alignment. “ ” denotes the mismatch of ranking order.
['[BOLD] Datasets', '[BOLD] Methods', '[BOLD] SA  [BOLD] PPL', '[BOLD] SA  [BOLD] Rank', '[BOLD] Alignment  [BOLD] AER', '[BOLD] Alignment  [BOLD] Rank']
Ngrad is better than Attn in terms of SA but the result is opposite in terms of AER. Since the evaluation criteria of SA and AER are different, it is reasonable that their evaluation results are different. This finding is in line with the standpoint by Jacovi and Goldberg However, it is observed that the ranking by SA is consistent on all three tasks but that by AER is highly dependent on different tasks.