Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering
Table 2: Effect of order of template, wh word and question mark on downstream QA performance.
['Template data', 'EM', 'F1']
Effect of template components: We evaluate the effect of individual template components on downstream QA performance. Wh template methods improve largely over the simple cloze templates. “Wh + B + A + ?” performs best among the template-based methods, as having the Wh word at the beginning most resembles the target SQuAD domain and switching the order of Fragment B and Fragment A may force the model to learn more complex relationships from the question. We additionally test the effect of the wh-component and the question mark added at the end of the sentence. Using the same data as “Wh + B + A + ?” but removing the wh-component results in a large decrease in performance. We believe that this is because the wh-component signals the type of possible answer entities, which helps narrow down the space of possible answers. Removing the question mark at the end of the template also results in decreased performance, but not as large as removing the wh-component. This may be a result of BERT pretraining which expects certain punctuation based on sentence structure. We note that these questions may not be grammatical, which may have an impact on performance. Improving the question quality makes a difference in performance as seen from the jump from cloze-style questions to template questions. The ablation studies suggest that a combination of question relevance, though matching entities, and question formulation, as described above, determine downstream performance. Balancing those two components is an interesting problem and we leave improving grammaticality and fluency through means such as language model generation for future experiments. Using the most-common wh word by grouping named entities into 5 categories according to Lewis et al. These results suggest that information about named entity type signaled by the wh-word does provide important information to the model but further information beyond wh-simple does not improve results significantly.