Cross-Lingual Training for Automatic Question Generation
Table 3: Human evaluation results as well as inter-rater agreement (column “Kappa”) for each model on the Hindi test set. The scores are between 0-100, 0 being the worst and 100 being the best. Best results for each metric (column) are in bold. The three evaluation criteria are: (1) syntactic correctness (Syntax), (2) semantic correctness (Semantics), and (3) relevance to the paragraph (Relevance).
['Model', 'Syntax Score', 'Syntax Kappa', 'Semantics Score', 'Semantics Kappa', 'Relevance Score', 'Relevance Kappa']
We conduct a human evaluation study comparing the questions generated by the Transformer and CLQG+parallel models. We randomly selected a subset of 100 sentences from the Hindi test set and generated questions using both models. We presented these sentence-question pairs for each model to three language experts and asked for a binary response on three quality parameters namely syntactic correctness, semantic correctness and relevance. Although we perform comparably to the baseline model on syntactic correctness scores, we obtain significantly higher agreement across annotators using our cross-lingual model. Our cross-lingual model performs significantly better than the Transformer model on “Relevance” at the cost of agreement. On semantic correctness, we perform signficantly better both in terms of the score and agreement statistics.