Reinforced Extractive Summarization with Question-Focused Rewards
Table 3: Train/valid accuracy and R-2 F-scores when using varying numbers of QA pairs (K=1 to 5) in the reward func.
['[EMPTY]', '[BOLD] K1', '[BOLD] K2', '[BOLD] K3', '[BOLD] K4', '[BOLD] K5']
The summaries are encouraged to contain comprehensive content useful for answering all questions. When more QA pairs are used (K1→K5), we observe that the number of answer tokens has increased and almost doubled: 23.7K (K1)→50.3K (K5) for entities as answers, and 7.3K→13.7K for keywords. The enlarged answer space has an impact on QA accuracies. When using entities as answers, the training accuracy is 34.8% (Q5) and validation is 15.4% (Q5), and there appears to be a considerable gap between the two. In contrast, the gap is quite small when using keywords as answers (27.5% and 21.9% for Q5), suggesting that using sentence root words as answers is a more viable strategy to create QA pairs.