MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants
Table 3: Accuracy on the test set and on script/text-based questions (accscr, acctxt) on MCScript. The maximum per column is printed in bold.
['[EMPTY]', '[BOLD] \xa0acc', '[BOLD] acc [ITALIC] scr', '[BOLD] acc [ITALIC] txt']
Comparison to MCScript. Since the same models were used for MCScript, a comparison of their performance on both datasets is possible. As can be seen, the performance of all three models is worse on MCScript2.0, showing that the dataset is generally more challenging. In contrast to MCScript, script-based questions in MCScript2.0 are clearly harder to answer than text-based questions: All models perform worse on script-based questions as compared to text-based questions. In comparison to MCScript, the performance of TriAN is 12% lower. This indicates that the new mode of question collection and the answer selection via adversarial filtering resolve some of the difficulties with MCScript. Question Types. It is clear that when questions are most challenging for all models. The logistic classifier performs almost at chance level.