Easy-to-Hard: Leveraging Simple Questions for Complex Question Generation
Table 6: Human evaluation results in both scenarios.
['[BOLD] w/ sub- questions', 'CoG2Q + data', '[BOLD] Correctness 0.5242', '[BOLD] Naturalness 3.24']
So as to complement the automatic evaluations, we conduct a user study to compare the data augmentation method with our models that explicitly leverage (pseudo) sub-questions. Four human annotators are asked to judge a generated question after seeing the corresponding query-graph and ground-truth complex questions. We randomly sample 100 examples for each scenario with sub-questions or pseudo sub-questions. The generated complex questions are judged based on correctness and naturalness. We instruct the annotators to count each relation or two relations connected with a CVT node as a single constraint. Correctness is defined as the fraction of constraints that have been expressed correctly, and naturalness is a score ranging between 1 and 5. In both scenarios, our methods outperform the data augmentation counterpart in terms of correctness, indicating that they can more effectively leverage the (pseudo) sub-questions. However, they do not achieve much higher naturalness scores. This is probably because, although directly copying words from human written (pseudo) sub-questions may contribute to higher naturalness, the fusion of copied words and generated words is prone to unnatural connections.