What If We Simply Swap the Two Text Fragments?A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks
Table 1: Swapping evaluation results on several NLI models. Swap-⋆ denotes the swapped data sets. Diff-⋆ denotes the performance drop of evaluated accuracy from the original data set to the swapped data set. We can observe the performance drop in many cases, some of which are significant. These results may indicate that these methods have been over-estimated in the ability in predicting the relationship of sentence pairs at the semantic level.
['Model', 'Label', 'Dev', 'Swap-Dev', 'Diff-Dev', 'Test', 'Swap-Test', 'Diff-Test']
We can observe the performance drop in many cases, some of which are significant. Therefore, following our previous argument, the results, unfortunately, indicate the potential incompetence of some of these evaluated models in capturing the semantic-level information despite these models’ impressive performances with the original evaluation metric. One may argue that the neutral pairs may not remain neutral after the two text fragments swapped, especially when the hypothesis is more specific than premise. For example, with a premise “I bought books”, several hypothesis can be generated to change the NLI label, such as “I bought 5 books”, “I bought books on history”, or “I bought books in the bookstore near campus”. However, through this paper, we assume the label preserves in most cases.