Evaluating Explanation Methods for Neural Machine Translation
Table 6: The PPL and Ranking Order comparison between two fidelity-based metric instantiations (Base and SA) on three WMT datasets. “ ” denotes the mismatch of ranking order.
['[BOLD] Datasets', '[BOLD] Methods', '[BOLD] Base  [BOLD] PPL', '[BOLD] Base  [BOLD] Rank', '[BOLD] SA  [BOLD] PPL', '[BOLD] SA  [BOLD] Rank']
S4SS3SSS0Px2 Results on WMT With the analysis of effects on various sample sizes, we choose a sample size of 1 million for the following scaling experiments. We can see that the order Pd > Ngrad > Attn > Wgrad evaluated by SA still remains unchanged on these three datasets as before. One can observe that the ranking order under the baseline doesn’t agree with SA on WMT De⇒En and Zh⇒En . Since the baseline yields in high PPL due to the mismatch we mentioned in section §3.3 ,in this case, we tend to trust the evaluation results from SA that achieves lower PPL leading to better fidelity.