What is this Article about? Extreme Summarization with Topic-aware Convolutional Neural Networks
Table 10: System ranking according to human judgments and QA-based evaluation for the Newsroom-Abs dataset.
['Models', 'Score', 'QA']
We did not include summaries from the lead, or ConvS2S+Copy as they were significantly inferior to other models. Based on the oracle extracts, participants can answer 41.31% of the questions correctly. Summaries generated by T-ConvS2S, PtGen, and ConvS2S provide answers to 23.28%, 20.98%, and 11.97% of the questions, respectively. ext-oracle performs best on both QA and judgment elicitation evaluations. However, we should point out that ext-oracle has the advantage of selecting the best set of sentences (as determined by ROUGE) without any length constraints. Consequently, summaries generated by ext-oracle tend to be longer with 38.84 words on average Summaries generated by PtGen, ConvS2S, and T-ConvS2S contain 23.72, 19.41 and 20.66 words, respectively, while gold summaries contain 23.26 words. Perhaps unsurprisingly T-ConvS2S, which was slightly lagging behind PtGen on R1 and RL scores, performs better than both PtGen and ConvS2S in terms of correctly answering questions. The QA evaluation shows that T-ConvS2S is able to generate informative summaries with pertinent information embedded in the appropriate context. Pairwise differences between systems are all statistically significant (p<0.01) with the exception of T-ConvS2S and PtGen, and, PtGen and ConvS2S.