Building an Evaluation Scale using Item Response Theory
Table 3: Comparison of Fleiss’ κ scores with scores from SNLI quality control sentence pairs.
['[BOLD] Fleiss’  [ITALIC] κ', '[BOLD] 4GS', '[BOLD] 5GS', '[BOLD] Bowman et al. 2015']
The decrease in agreement is particularly large with regard to contradiction. This could occur for a number of reasons. Recognizing entailment is an inherently difficult task, and classifying a correct label, particularly for contradiction and neutral, can be difficult due to an individual’s interpretation of the sentences and assumptions that an individual makes about the key facts of each sentence (e.g. coreference). It may also be the case that the individuals tasked with creating the sentence pairs on AMT created sentences that appeared to contradict a premise text, but can be interpreted differently given a different context.