Invariant Rationalization
Table 1: Results on the synthetic IMDB dataset. The last column is the percentage of testing examples with the injected punctuation selected as a part of the rationales. The best test results are bolded.
['[EMPTY]', 'Dev Acc', 'Test Acc', 'Bias Highlighted']
Imdb: As we can see, Rnp selects the injected punctuation in 78.24% of the testing samples, while InvRat, as expected, does not highlight any. Moreover, because Rnp relies on these injected punctuation, whose probabilistic distribution varies drastically between training set and test set, its generalizability is poor, which leads to low predictive accuracy on the testing set. Specifically, there is a large gap of around 15% between the test performance of Rnp and the proposed InvRat. It is worth pointing out that, by the dataset construction, 3Player will obviously fail by including all punctuation as rationales. This is because otherwise, the complement predictor will have a clear clue to guess the predicted label. Thus, we exclude 3Player from the comparison.