Hypothesis Only Baselines in Natural Language Inference
Table 2: NLI accuracies on each dataset. Columns ‘Hyp-Only’ and ‘MAJ’ indicates the accuracy of the hypothesis-only model and the majority baseline. |Δ| and Δ% indicate the absolute difference in percentage points and the percentage increase between the Hyp-Only and MAJ. Blue numbers indicate that the hypothesis-model outperforms MAJ. In the right-most section, ‘Baseline’ indicates the original baseline on the test when the dataset was released and ‘SOTA’ indicates current state-of-the-art results. MNLI-1 is the matched version and MNLI-2 is the mismatched for MNLI. The names of datasets are italicized if containing ≤10K labeled examples.
['Dataset', '[BOLD] DEV Hyp-Only', '[BOLD] DEV MAJ', '[BOLD] DEV  \\centering |Δ| \\@  [ITALIC] add@ [ITALIC] centering', '[BOLD] DEV  \\centering Δ \\@  [ITALIC] add@ [ITALIC] centering%', '[BOLD] TEST Hyp-Only', '[BOLD] TEST MAJ', '[BOLD] TEST  \\centering |Δ| \\@  [ITALIC] add@ [ITALIC] centering', '[BOLD] TEST Δ%', 'Baseline', 'SOTA']
Our goal is to determine whether a hypothesis-only model outperforms the majority baseline and investigate what may cause significant gains. In such cases a hypothesis-only model should be used as a stronger baseline instead of the majority class baseline. For all experiments except for JOCI, we use each NLI dataset’s standard train, dev, and test splits. Across six of the ten datasets, our hypothesis-only model significantly outperforms the majority-baseline, even outperforming the best reported results on one dataset, recast SPR. This indicates that there exists a significant degree of exploitable signal that may help NLI models perform well on their corresponding test set without considering NLI contexts. The largest relative gains are on human-elicited models where the hypothesis-only model more than doubles the majority baseline.