Evaluating Explanation Methods for Neural Machine Translation
Table 5: Evaluating four explanation methods on 3 different scenarios Real-Decode (R-Dec), Golden-Data (Golden) and Teacher-Forcing Decode (T-Dec)) for Transformer over IWSLT De⇒En task.
['[BOLD] Methods', '[BOLD] R-Dec', '[BOLD] Golden', '[BOLD] T-Dec']
, our metric instantiations are trained and evaluated under the same scenario, where ct used to extract relevant words is obtained from gold data and its label f(ct) is the prediction from NMT f, namely Teacher Forcing Decode. To examine the robustness of our metric, we apply the trained metric to two different scenarios: real decoding scenario (Real-Decode) where both ct and its label f(ct) are from the NMT output; and golden data scenario (Golden-Data) where both ct and its label are from golden test data. To our surprise, the results in Real-Decode are even better than those in the matched Teacher Forcing Decode scenario. One possible reason is that the labels generated by a NMT system in the Real-Decode tend to be high-frequency words, which leads to better PPL. In contrast, our metric instantiation in the Golden-Data results in much higher PPL due to the mismatch between training and testing. The performance of experimenting training and testing in the same scenario like Golden-Data can be experimented in future works, however, it’s not the focus of this paper.