[
Table 10: MultiNLI-mm linguistic error analysis (accuracy %), comparing our HBMP results to the InferSent Conneau et al. (2017) results (our implementation). Number of sentence pairs with the linguistic label in brackets after the label name.
['[EMPTY]', 'Entailment  [BOLD] HBMP', 'Entailment  [BOLD] InferSent', 'Contradiction  [BOLD] HBMP', 'Contradiction  [BOLD] InferSent', 'Neutral  [BOLD] HBMP', 'Neutral  [BOLD] InferSent']
The MultiNLI annotation set makes it possible to conduct a detailed analysis of different linguistic phenomena when predicting inferential relationships. We use this to compare our model to InferSent with respect to the type of linguistic properties that are present in the given sentence pairs. The analysis shows that our HBMP model outperforms InferSent with antonyms, coreference links, modality, negation, paraphrases and tense differences. It also produces improved scores for most of the other categories in entailment detection. InferSent gains especially with conditionals in contradiction and in the word overlap catehory for entailments. This seems to suggest that InferSent relies a lot on matching words to find entailment and specific constructions indicating contradictions. HBMP does not seem to use word overlap as an indication for entailment that much and is better on detecting neutral sentences in this category. This outcome may indicate that our model works with stronger lexical abstractions than InferSent. However, due to the small number of examples per annotation category and small differences in the scores in general, it is hard to draw reliable conclusions from this experiment.