Adversarial Filters of Dataset Biases
Table 4: Dev accuracy (%) on original and AFLite-filtered MNLI-matched and QNLI. The -PartialInput baselines show models trained on only Hypotheses for MNLI instances and only Answers for QNLI. Δ indicates the difference in performance between the full model and the model trained on AFLite-filtered data.
['Model', 'MNLI  [ITALIC] D', 'MNLI  [ITALIC] D( [ITALIC] ϕRoBERTa)', 'Δ']
We evaluate the performance of another large-scale NLI dataset multi-genre NLI Since RoBERTa resulted in the largest drops in performance across the board in SNLI, we only experiment with RoBERTa as adversary for MultiNLI and QNLI. While RoBERTa achieves over 90% on both original datasets, its performance drops to 66.2% for MultiNLI and to 77.7% for QNLI on the filtered datasets. Similarly, partial input baseline performance also decreases substantially on both dataset compared to their performance on the original dataset. Overall, our experiments indicate that AFLite consistently results in reduced accuracy on the filtered datasets across multiple language benchmark datasets, even after controlling for the size of the training set.