Option Comparison Network for Multiple-choice Reading Comprehension
Table 2: Experimental results. The best results in each group are in bold, and those better than Amazon Mechanical Turker are underlined.
['Model', 'Pre-training', 'RACE-M', 'RACE-H', 'RACE']
as the skimmer (Sec. From the results we can observe that: (1) Our model outperforms the baselines significantly, indicating the effectiveness of our model. (2) Our ensemble model with BERT\textscLARGE as the skimmer surpasses Amazon Mechanical Turker on the whole dataset and the margin on the RACE-H subset is significantly large. Moreover, OCN\textscLARGE also outperforms Amazon Mechanical Turker without ensembling. All these results indicate that our model has learned certain reasoning skills. (3) There is still a large gap between human ceiling performance and our model’s performance. We believe this is because our model still struggles in complex reasoning as expected. (4) All the models using pre-trained contextualized representations (GPT Radford et al. is a promising research direction for learning semantics from unsupervised data.