How to Ask Better Questions? A Large-Scale Multi-Domain Dataset for Rewriting Ill-Formed Questions
Table 4: Summary of manual annotations for instances sampled from the DEV and TEST portions of the MQR dataset. “Quality” are the average quality scores, broken down into three aspects. “Semantic Equivalence” is the percentage of question pairs in which the ill-formed and well-formed questions are semantically equivalent. The scores are averages of binary scores across both annotators.
['[BOLD] Quality Ill-formed', '[BOLD] Quality Ill-formed', '[BOLD] Quality Ill-formed', '[BOLD] Quality Well-formed', '[BOLD] Quality Well-formed', '[BOLD] Quality Well-formed', '[BOLD] Semantic  [BOLD] Equivalence']
We summed up the binary scores from two annotators. There are clear differences between ill-formed and well-formed questions. Ill-formed question are indeed ill-formed and well-formed questions are generally of high quality. The average score over three aspects improves by 45 points from ill-formed to well-formed questions. Over 90% of the question pairs possess semantic equivalence, i.e., they do not introduce or delete information. Therefore, the vast majority of rewrites can be performed without extra information. When the two annotators disagree, a judge makes a final decision. According to the annotations, the GEC method slightly improves the question quality scores. However, the GEC model is good at preserving question semantics.