LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Table 4: Transfer learning results (accuracy%).
['[BOLD] Model', '[BOLD] Dev', '[BOLD] Test']
We conduct a set of transfer learning experiments to understand the degree of overlap in terms of necessary knowledge for solving problems in our dataset and existing datasets. In particular, we first fine-tune the RoBERTa model on a source dataset, before fine-tuning the model on LogiQA. If the required knowledge is similar, the model performance is expected to increase. RACE and COSMOS are adopted as the source datasets. The former tests English reading skills while the latter tests commonsense knowledge. The performance of RoBERTa trained on RACE is even close to the random guess baseline. In addition, further fine-tuning on LogiQA leads to improvements over the source-trained baselines, but the resulting models do not outperform a model trained only on LogiQA.