Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Table 4: Pearson correlation coefficients between QAGS scores with varying number of questions and human judgments of correctness for summarization datasets. The correlation increases with the number of questions used, but with decreasing marginal benefit.
['# Questions', 'CNN/DM', 'XSUM']
Next, we investigate the correlation with human judgments when varying the number of questions used. We observe a large increase when moving from 10 to 20 questions, and a smaller increase from 20 to 50 questions, indicating decreasing marginal benefit moving beyond 50 questions. With just 5 questions, QAGS still substantially outperforms other automatic metrics, indicating its robustness.