A Logic-Driven Framework for Consistency of Neural Models
Table 2: Inconsistencies (%) of models on our 100k evaluation dataset. Each number represents the average of three random runs. Models are trained using 5% and 100% of the train sets. SNLI+MultiNLI2: finetuned twice. ρS and τS: symmetry consistency violations. ρT and τT: transitivity consistency violations.
['Config', '5%  [ITALIC] ρS', '5%  [ITALIC] τS', '5%  [ITALIC] ρT', '5%  [ITALIC] τT', '100%  [ITALIC] ρS', '100%  [ITALIC] τS', '100%  [ITALIC] ρT', '100%  [ITALIC] τT']
We see that both LSTM and BERT models have symmetry consistency violations, while the transitivity consistency has lower violations. Surprisingly, the LSTM model performed on par with BERT in terms of symmetry/transitivity consistency; stronger representations does not necessarily mean more consistent models.