LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Table 5: Performance of different length (accuracy%).
['[BOLD] Length', '[BOLD] (0,100]', '[BOLD] (100,150]', '[BOLD] (150,200]', '[BOLD] (200,+∞)']
We measure the accuracy of RoBERTa against the input size. In particular, the number of words in the paragraph, the question and the candidate answers are added together as the length of a test instance. Interestingly, the model performances are not negatively associated with the input size, which is different from most NLP benchmarks. This shows that the level of challenge in logical reasoning can be independent of the input verbosity.