Learning to Deceive with Attention-Based Explanations
Table 3: Accuracy of various classification models along with their attention mass (A.M.) on impermissible tokens I, with varying values of the loss coefficient λ. The first row for each model class represents the case when impermissible tokens I for the task are deleted/anonymized. For most models, and tasks, we can severely reduce attention mass on impermissible tokens while preserving original performance (λ=0 implies no manipulation).
['Model', '[ITALIC] λ', 'I', 'Occupation Pred. Acc.', 'Occupation Pred. A.M.', 'Gender Identify Acc.', 'Gender Identify A.M.', 'SST + Wiki Acc.', 'SST + Wiki A.M.', 'Ref. Letters Acc.', 'Ref. Letters A.M.']
For the classification tasks, we experiment with the loss coefficient λ∈{0,0.1,1}. In each experiment, we measure the (i) attention mass: the sum of attention values over the set of impermissible tokens averaged over all the examples, and (ii) test accuracy. During the course of training (i.e. after each epoch), we arrive at different models from which we choose the one whose performance is within 2% of the original accuracy and provides the greatest reduction in attention mass on impermissible tokens. Across most tasks, and models, we find that our manipulation scheme severely reduces the attention mass on impermissible tokens compared to models without any manipulation (i.e. when λ=0). This reduction comes at a minor, or no, decrease in task accuracy. Note that the models can not achieve performance similar to the original model (as they do), unless they rely on the set of impermissible tokens. This can be seen from the gap between models that do not use impermissible tokens ( I ✗) from ones that do ( I ✓).