LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Table 3: Main results on LogiQA (accuracy%).
['[BOLD] Category', '[BOLD] Model', 'LogiQA  [BOLD] Dev', 'LogiQA  [BOLD] Test', 'Chinese LogiQA  [BOLD] Dev', 'Chinese LogiQA  [BOLD] Test']
We randomly split the dataset, using 80% for training, 10% for development and the remaining 10% for testing. In particular, the human performance is 86.00% and the ceiling performance is 95.00%, which shows that the difficulty level of the dataset is not high for human testees. In contrast, all of the algorithmic models perform significantly worse than human, demonstrating that the methods are relatively weak in logical reasoning reading comprehension. In addition, results on the Chinese dataset are on the same level compared with those on the English dataset.