Multi-Document Abstractive Summarization Using ILP based Multi-Sentence Compression
Table 2: Manual evaluation by 10 evaluators on Informativeness (Inf) and Linguistic Quality (LQ) of summaries. Average Log-likelihood scores (Avg.LL) from parser are also shown.
['[BOLD] Type', '[BOLD] Inf', '[BOLD] LQ', '[BOLD] Avg.LL']
The four summaries provided to the evaluators are human-written summary (one summary collected randomly from four model-summaries per cluster), extractive summary (DPP), abstractive summary generated using MSC (MSC) and abstractive summary generated using our ILP based method (ILPSumm). We asked each evaluator to complete 10 such tasks, each containing four summaries as explained above. We normalize ratings of different evaluators to the same scale. According to the judges, the linguistic quality of ILPSumm (3.63) is significantly better than that of MSC (2.83). Further, our summaries (ILPSumm) are more informative than DPP (3.90) and MSC (3.78). DPP is extractive in nature, hence linguistically, the sentences are generally more readable. A language model assigns probabilities to sequence of words; in contrast, the confidence score of a parser assigns probabilities to grammatical relations. The values (the lower the magnitude, the better) are shown in the column Avg. LL. Avg. LL obtained by ILPSumm (-180.76) is better than that obtained by MSC (-210.02), indicating that the language model based linguistic quality estimation helps generate more readable summaries than the MSC method.