Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
Table 8: Accuracy of Bi-LSTM classifier trained on hypotheses only
['Train/Test', 'Original', 'RP', 'RH', 'RP & RH']
To further isolate this effect, Bi-LSTM trained on SNLI hypotheses Note that this combined dataset consists of five variants of each original premise-hypothesis pair. Of these five pairs, three consist of the same hypothesis sentence, each associated with different truth value given the respective premise. Using these hypotheses only would provide conflicting feedback to a classifier during training, thus causing the drop in performance. Further, we notice that the gain of the latter over majority class baseline comes primarily from the original data, as the same model retrained only on RP and RH data experiences a further drop of 11.6% in accuracy, performing worse than just choosing the majority class at all times.