A Logic-Driven Framework for Consistency of Neural Models
Table 8: Symmetry/Transitivity inconsistencies (%) for models using 1%, 5%, 20%, and 100% training data. Each number represents the average of three random runs. SNLI+MultiNLI2: BERTbase finetuned twice for fair comparison. SNLI/MultiNLI column: accuracies on corresponding text sets. M: mirrored labeled examples. U: unlabeled instance pairs. T: unlabeled instance triples.
['Train', '1% SNLI', '1% MultiNLI', '1%  [ITALIC] ρS', '1%  [ITALIC] τS', '1%  [ITALIC] ρT', '1%  [ITALIC] τT', '5% SNLI', '5% MultiNLI', '5%  [ITALIC] ρS', '5%  [ITALIC] τS', '5%  [ITALIC] ρT', '5%  [ITALIC] τT']
a.3.2 Results Note that the accuracies of baselines finetuned twice are slightly better than models only finetuned once, while their symmetry/transitivity consistencies are roughly on par. We found such observation is consistent with different finetuning hyperparameters (\egwarming, epochs, learning rate).