Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
Table 9: Accuracy of models trained to differentiate between original and revised data
['Model', 'IMDb', 'SNLI/RP', 'SNLI/RH']
One reasonable concern might be that our models would simply distinguish whether an example were from the original or revised dataset and thereafter treat them differently. The fear might be that our models would exhibit a hypersensitivity (rather than insensitivity) to domain. BERT identifies original reviews from revised reviews with 77.3% accuracy. In case of NLI, BERT and Naïve Bayes perform roughly within 3 pts of the majority class baseline (66.7%) whereas SVM performs substantially worse.