Conditional Neural Generation using Sub-Aspect Functions for Extractive News Summarization
Table I: ROUGE and Human evaluation scores of oracle summaries built on BertScore and ROUGE.
['[BOLD] Similarity Evaluation', '[BOLD] Score']
We observed that the oracle summaries generated through semantic similarity is different from those chosen from traditional n-gram overlap. To further evaluate the effectiveness of this oracle construction approach, we conducted two assessments. First, we calculated their ROUGE scores with the gold summary. Next, we conducted 2 human evaluation experiments. The evaluation score represents the likelihood of a higher ranking, and is normalized to range [0,1]. For each sentence of gold summary, questions were related to key information like event and name entities. Then human annotators were asked to answer these questions given an oracle summary. The accuracy of answers is regarded as the evaluation score. Moreover, we constructed some extended questions in which the answer can only be answered with comprehension of the full summary.