What If We Simply Swap the Two Text Fragments?A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks
Table 2: “Stress Test” results for different percentage of training text fragments to be swapped. S⋆ denotes the percentage of text fragments are randomly swapped during training (Only neutral and Contrdiction pairs can be swapped). Accuracies shown on both genre-matched and mismatched categories for each stress test. R⋆ denotes the ratio of results over the case when no pairs are swapped (S0%). Devi (last column) denotes the overall deviation of R⋆ from 100% within each test.
['Model', 'Test', 'Cat', 'S0%', 'S25%', 'R25%', 'S50%', 'R50%', 'S75%', 'R75%', 'S100%', 'R100%', 'Devi']
S⋆ reports scores with different training data, and R⋆ reports the relative changes of the corresponding case (ratio of S⋆ over S0%). We also calculated the deviation for each stress test, which indicates the robustness of the model to confounding factors (the smaller, the better). The deviation is calculated as the sum of the squared error between R⋆ and 100% and is reported in the last column. Overall, DGA is least affected by the spurious signals, which is consistent with our results in the previous section. For other methods, InferSent and ADV show robustness except on Antonymy test, KIM shows robustness except on Negation test (which is also consistent with the findings in the previous section), and ESIM shows robustness except in Word Overlap and Spelling Error case.