At Which Level Should We Extract? An Empirical Study on Extractive Document Summarization
Table 4: Human evaluation results. Unnecessity and Redundancy are reported as occurrence frequency, and lower is better. Readability and Overall are reported as ranking, and lower is better.
['[EMPTY]', 'BERT-SENT', 'SSE']
We compare the SSE with BERT-SENT. As shown in the results, SSE performs better than BERT-SENT for both redundancy and unnecessity. The frequency of having these issues drops 20% and 17% respectively. Thus the overall quality of SSE is also better than BERT-SENT. As shown in results, the BERT-SENT is always ranked as the best since the sentences are fully extracted. The readability of extracted sub-sentential units is slightly worse than the full sentences. We also manually checked the output of SSE whose readability is worse. We found that there are two reasons: 1) the sub-sentence is fragmented which affects the readability; 2) the sub-sentence is wrongly extracted due to the error of the constituency parser. Therefore, we hope that the readability of SSE could be improved if we can: 1) design better sub-sentential unit extraction algorithm; 2) have an even better syntactic parser.