Collecting Diverse Natural Language Inference Problemsfor Sentence Representation Evaluation
Table 3: NLI accuracies on test data. Columns correspond to each semantic phenomena and rows correspond to the model used. Columns are ordered from larger to smaller in size, but the last three (VC, MV, VN) are separated since they fall under lexico-syntactic inference. (update) refers to a model that was initialized with pre-trained parameters and then re-trained on the corresponding recast data. (fixed) refers to a model that was trained and then evaluated on these data sets. Bold numbers in each column indicate which settings were responsible for the highest accuracy on the specific recast dataset.
['[width=9em]ModelRecast Data', 'NER', 'EF', 'RE', 'Puns', 'Sentiment', 'GAR', 'VC', 'MV', 'VN']
Even though we categorize VerbNet, MegaVeridicality, and VerbCorner as lexicosyntatic inference, we train and evaluate models separately on these three datasets because we use different strategies to individually recast them. When evaluating NLI models, our baseline is the maximum between the accuracies of the hypothesis-only model and the majority class label (MAJ). In six of the eight recast datasets that we use to train our models the hypothesis-only model outperforms MAJ. The two datasets where the hypothesis-only model does not outperform MAJ are Sentiment and VN, each of which contain less than 10K examples. We do not train on GAR because of its small size.