TGSum: Build Tweet Guided Multi-Document Summarization Dataset
Table 5: Manual evaluation of TGSum’s quality.
['Metrics Informativeness', 'Metrics Good', 'TGSum 0.66', 'TWEET 0.60']
We manually evaluate the sentence quality in the generated reference summaries. Two metrics are used, i.e., informativeness and readability. Since TGSum serves for learning-based summarization models, we do not consider the coherence of an entire summary. For each metrics, a summary sentence is classified into three levels, namely good, OK and bad. Linked tweets are also evaluated for comparison. As seen, most sentences in the reference summaries are both informative and readable. The former owes to the saliency of most linked tweets, while the candidate sentence selection strategy brings the nice readability. Since a generated reference is the ROUGE upper bound measured by the set of linked tweets, a small amount of noised tweets will be excluded. Thus the number of unimportant sentences in reference summaries is much smaller than that in linked tweets. A bad case we find is the cluster about “#BBCBizQuiz”. All the tweets are questions. As a result, the reference summary fails to locate the salient sentences. With regard to readability, a large part of linked tweets are merely regarded as OK because they are not the complete sentences at all. By contrast, sentences in reference summaries are usually formal and readable. Quite a few summary sentences marked as OK are due to the splitting errors. For instance, sometimes a section title is attached to a sentence. Meanwhile, a pronoun at the beginning of a sentence brings the co-reference ambiguity problem.