LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Table 6: Overlap ratio (%) against the model type.
['[BOLD] Model', '[BOLD] Overlap Ratio', '[BOLD] Accuracy(%)']
We aim to understand a bias of models in selecting the candidate answers that have the best surface matching with the paragraph. To this end, we calculate the unigram overlap between each candidate answer and the given paragraph for each problem, and mark the best-matching candidate. We report the “Overlap Ratio” by calculating the accuracy between model prediction and the best-matching candidate. As can be seen, the gold-standard output has an accuracy of 28.37%, whilst all of the models give accuracies above this number, which shows a tendency of superficial matching. In particular, the word matching method gives an accuracy of 100% due to its mechanism. RoBERTa gives the lowest matching accuracy, showing that it relies the least on lexical patterns. We additionally measure the accuracy of the models on the “correct” instances according to best matching. RoBERTa still outperforms the other models, demonstrating relative strength in logical reasoning.