Generating Distractors for Reading Comprehension Questions from Real Examinations
Table 3: Human evaluation results. Note that we allow annotators to choose more than one options if the generated outputs are accidentally the same or very semantically similar, therefore, the total number of selected options (552) is larger than the total number of annotated questions (540).
['[EMPTY]', 'Annotator 1', 'Annotator 2', 'Annotator 3', '# Selected']
For comparison, we count how many times of individual pipelines (the ground truth and three compared methods) are successful in confusing the annotators, i.e. their distractors are selected as answers. We give each annotator 60 articles, and 3 questions per article. We find that the ground truth distractors (i.e. by “Human”) have the strongest capability to confuse the annotators. Among the compared automatic methods, our model performs the best, while Seq2Seq performs the worst, which is a consistent conclusion as drawn from the previous section.