Analyzing representations
Table 4: Percentage of entailments split by overlap rate of words in SNLI.
['Top', 'Entailment', 'Neutral', 'Contradiction']
We carried out an analysis of the SNLI dataset to determine if the overlap heuristic is ecologically valid in it. First, we observed anecdotally that indeed several contradictory sentence pairs have no overlap in words. For example, a contradictory sentence pair in SNLI is: Premise: Several people are trying to climb a ladder in a tree. Hypothesis: People are watching a ball game. CONTRADICTION To quantitatively verify this observation , we ranked all the sentence pairs in SNLI by overlap rate: \# of overlap wordstotal \# of words(in non-increasing order). We then considered the top X sentences with highest overlap for different Xs. However, we found that as the word overlap in the sentences increases, the percentage of contradictions drops. When considering only the top 1000 sentence pairs for overlap, we found that 91.5% of them have entailment or neutral labels, with only the remaining 8.5% having a contradiction label.