Learning to Deceive with Attention-Based Explanations
Table 4: Performance of sequence-to-sequence models and their attention mass (A.M.) on impermissible tokens I, with varying values of the loss coefficient λ. Similar to classification tasks, we can severely reduce attention mass on impermissible tokens while retaining original performance. All values are averaged over five runs.
['Attention', '[ITALIC] λ', 'Bigram Flip Acc.', 'Bigram Flip A.M.', 'Sequence Copy Acc.', 'Sequence Copy A.M.', 'Sequence Reverse Acc.', 'Sequence Reverse A.M.', 'En → De MT BLEU', 'En → De MT A.M.']
To measure performance, we use token-by-token accuracy for synthetic tasks, and BLEU score for English to German MT. We also notice that the models with manipulated attention (i.e. deliberately misaligned) outperform models with none or uniform attention. This suggests that attention mechanisms add value to the learning process in sequence-to-sequence tasks which goes beyond their usual interpretation as alignments.