Guiding Extractive Summarization with Question-Answering Rewards
Table 4: Question-answering accuracies using different types of QA pairs (ROOT, SUBJ/OBJ, NER) and different source input (NoText, QASumm+NoQ, GoldSumm, and FullText) as the basis for predicting answers.
['[EMPTY]', '[BOLD] NoText  [BOLD] Train', '[BOLD] NoText  [BOLD] Dev', '[BOLD] NoText  [BOLD] Gap', '[BOLD] QASumm+NoQ  [BOLD] Train', '[BOLD] QASumm+NoQ  [BOLD] Dev', '[BOLD] QASumm+NoQ  [BOLD] Gap', '[BOLD] GoldSumm  [BOLD] Train', '[BOLD] GoldSumm  [BOLD] Dev', '[BOLD] GoldSumm  [BOLD] Gap', '[BOLD] FullText  [BOLD] Train', '[BOLD] FullText  [BOLD] Dev', '[BOLD] FullText  [BOLD] Gap']
It outperforms the scenarios using FullText as the source input. This indicates that distilled information contained in a high-quality summary can be useful for answering questions, as searching for answers in a succinct summary can be more efficient than that in a full article. Moreover, we observe that the performance of QASumm+NoQ is in between NoText and GoldSumm for all answer types. The results suggest that extractive summaries with even modest ROUGE scores can prove useful for question-answering. Regarding different types of QA pairs, we find that the ROOT-type can achieve high QA accuracy when using NoText input. It suggests that ROOT answers can to some extent be predicted based on the question context. The NER-type QA pairs work the best for both GoldSumm and FullText, likely because the source texts contain necessary entities required to correctly answer those questions. We also find the SUBJ/OBJ-type QA pairs have the smallest gap between train/dev accuracies, despite that they have a large answer space. Based on the analysis we would suggest future work to consider using NER-based QA pairs as they encourage the summaries to contain salient source content and be informative.