HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference
Table 4: Model baselines on the proposed hard and easy test sets. For MultiNLI, we trained the models using matched dev sets as the valid set and reported the test accuracies on mismatched dev sets. ’UW+CMU’ refers to the adversarial set (Sec 2.4) detected by a neural based hypothesis-only classifier.
['[EMPTY]', 'Full', 'Easy', 'Hard']
We set up both pretrained and non-pretrained model baselines for the proposed adversarial datasets. For BERT, XLNet and RoBERTa, we concatenate the premise sentence and hypothesis sentence with [SEP] token as the input. For output classifier, we use a linear mapping to transform the vector at the position of [CLS] token at the last layer of these pretrained models to a normalized 3-element vector (using softmax) which represents the scores for each label.