Knowledge-Driven Distractor Generation for Cloze-style Multiple Choice Questions
Table 2: End-to-end comparison on test set. - means no ranking algorithm to evaluate and “ground truth” denotes the score of ground-truth distractors associated with each item.
['[BOLD] Method', '[BOLD] Human Evaluation Reliability', '[BOLD] Human Evaluation Plausibility', '[BOLD] Automatic Evaluation F1@3', '[BOLD] Automatic Evaluation P@1', '[BOLD] Automatic Evaluation P@3', '[BOLD] Automatic Evaluation R@3', '[BOLD] Automatic Evaluation MRR', '[BOLD] Automatic Evaluation NDCG@10', '[BOLD] Automatic Evaluation Semantic Similarity@3']
End-to-End Comparison. Despite the significantly reduced number of candidates, ranking methods with our candidate set generator can achieve much higher performance than with unstructured vocabulary in much shorter runtime. TM performs badly due to its naive path similarity ranking criterion. The results of ED are worst among all unsupervised methods while embedding based methods can even achieve comparable performance against LR+LM/ RF when provided with a high-quality candidate set. BERT ranks distractors using contextualized representation thus leading to lowest reliability according to human evaluation. LR+RF/LM achieves similar ranking performance yet obtain poorer reliability than CSG+DS since they only focus on the plausibility of selected distractors. CSG+DS, despite its relative simplicity, obtain consistent improvements over LR+RF/LM without two-stage cascaded training. We observe certain inconsistency between plausibility and automatic metrics of baselines, part of the reason may be that methods such as LR+RF/LM focus much on shallow feature patterns of ground-truth distractors and fail to unearth potential acceptable distractors. However, distractors generated by CSG+DS yield highest ranking measures while rated as most plausible by human annotators. Unsupervised methods work solely relying on the semantic similarity hence their reliabilities are generally lower than supervised ones, among which our DS turns out to be the most reliable. Exceptionally, EmbSim+CF gets higher reliability with WordNet, whose unreliable candidates get more chance to be eliminated by post-filtering than those in Probase.