A Sentence Compression Based Framework to Query-Focused Multi-Document Summarization
Table 6: Human evaluation on our multi-scorer based system, iiit06 (Best DUC system (ROUGE)), and lcc06 (Best DUC system (LQ)). Our system can synthesize more relevant content according to Pyramid (×100). We also examine linguistic quality (LQ) in Grammaticality (Gra), Non-redundancy (Non-Red), Referential clarity (Ref), Focus (Foc), and Structure and Coherence (Coh) like duc, each rated from 1 (very poor) to 5 (very good). Our system has better non-redundancy than iiit06 and is comparable to iiit06 and lcc06 in other metrics except grammaticality.
['[BOLD] System', '[BOLD] Pyr', '[BOLD] Gra', '[BOLD] Non-Red', '[BOLD] Ref', '[BOLD] Foc', '[BOLD] Coh']
We ask a professional annotator (who is not one of the authors, is highly experienced in annotating for various NLP tasks, and is fluent in English) to carry out a Pyramid evaluation on 10 randomly selected topics from the DUC 2006 task with gold-standard SCU annotation in abstracts. We further evaluate the linguistic quality (LQ) of the summaries for the same 10 topics in accordance with the measurement in \newciteduc. (Best DUC system (LQ)) Our system achieves a higher Pyramid score, an indication that it captures more of the salient facts. We also attain better non-redundancy than \newciteiiit06, meaning that human raters perceive less replicative content in our summaries. Scores for other metrics are comparable to \newciteiiit06 and \newcitelcc06, which either uses minimal non-learning-based compression rules or is a pure extractive system. However, our compression system sometimes generates less grammatical sentences, and those are mostly due to parsing errors. For example, parsing a clause starting with a past tense verb as an adverbial clausal modifier can lead to an ill-formed compression. Those issues can be addressed by analyzing k-best parse trees and we leave it in the future work.