Generating Synthetic Data for Neural Keyword-to-Question Models
Table 4. Manual K2Q evaluation results. The inter-rater agreement is measured using Cohen’s kappa score (Cohen, 1960). Highest scores are in boldface.
['[BOLD] Methods', '[BOLD] Relevance', '[BOLD] Grammar', '[BOLD] Wins']
As expected, the SDM method scores highest on grammar, since it retrieves existing questions from the corpus. However, it achieves a very low score on relevance, since it can only retrieve questions that have been asked before (i.e., exist in the corpus). As in the automatic evaluation results, the attention mechanism brings in substantial improvements over the simple Encoder-Decoder model (both in terms of relevance and grammar). As anticipated, the copying mechanism leads to large improvements in terms of relevance (+40.3%); at the same time, the grammar score of CopyNet is only marginally lower than that of AttNet.