Bringing Structure into Summaries: Crowdsourcing a Benchmark Corpus of Concept Maps
Table 1: Correlation of peer scores with manual responsiveness scores on TAC2008 topics 01-03.
['[BOLD] Peer Scoring', '[BOLD] Pearson', '[BOLD] Spearman']
In addition to the reliability studies, we extrinsically evaluated the annotations in the task of summary evaluation. For each of the 58 peer summaries in TAC2008, we calculated a score as the sum of the importance estimates of the propositions it contains. The results demonstrate that with both task designs, we obtain importance annotations that are similarly useful for summary evaluation as pyramid annotations or gold-standard summaries (used for ROUGE).