Knowledge-Driven Distractor Generation for Cloze-style Multiple Choice Questions
Table 1: Comparison of combinations of different choices of CSG and DS. - means no ranking.
['[BOLD] Instantiation CSG', '[BOLD] Instantiation DS', 'F1@3', 'P@1', 'P@3', 'R@3', 'MRR', 'NDCG@10', 'Semantic Similarity@3']
Combinations of CSG and DS. Without CSG, distractor selector trained with trivial negative examples is forced to select distractors from a rather large and noisy candidate set therefore the performance is clearly worse. We also find that combining CSG with DS yields consistent improvement by all metrics and the improvement is more significant for WordNet CSG, which is mainly because p(c|a) and p(d|c) in WordNet are partly biased due to the limited scale of corpus they are estimated on, hence the supervised training will lead to more performance gain. Pair/list-wise ranker achieve comparable performance mainly due to the binarized relevance score. Since named entities and common nouns mainly underpin Probase, DS with Probase CSG naturally get higher ranking scores on our dataset than its counterpart with WordNet CSG.