What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
Figure 9: Case Study of AddSent2Opt-Shuffle, we initialize the distractor as a empty sentence paddings and add the token from the AddSent2Opt-Shuffle distractor one by one to analyse the predicted probability change. The correct answer remains unchanged in this process.
['Probability', 'Answer', 'Distractor']
Case Study We can observe that: 1) Keyword matching plays an important role, for example, adding the word “number” to the distratcor option that matches the number “38.25” in the question significantly decreases the probability of the correct answer. 2) Keywords from the original distractor can also help the model identify the correct answer. For example, adding the word “counter” from the original distractor increases the probability for the correct answer. 3) Adding more matched words to the distratcor options can decrease the probability for the correct answer. 4) Stopwords can also influence the prediction.