ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Table 6: Average accuracy of each model using four different random seeds with only answer options as input, and the number of their common correct predictions.
['[BOLD] Model', '[BOLD] Val', '[BOLD] Test', '[BOLD] Number']
CEASY = (Cseed1GPT∩Cseed2GPT∩Cseed3GPT∩Cseed4GPT) (1) ∪(Cseed1GPT−2∩Cseed2GPT−2∩Cseed3GPT−2∩Cseed4GPT−2) ∪(Cseed1XLNet∩Cseed2XLNet∩Cseed3XLNet∩Cseed4XLNet) ∪(Cseed1RoBERTa∩Cseed2RoBERTa∩Cseed3RoBERTa∩Cseed4RoBERTa), CHARD = CTEST−CEASY, where Cseed1BERT denotes the set of data points which are predicted correctly by BERT\textscBASE with seed 1, and similarly for the rest. Finally, we get 440 data points from the testing set CTEST and we denote this subset as EASY set CEASY and the other as HARD set CHARD.