TGSum: Build Tweet Guided Multi-Document Summarization Dataset
Table 6: Summarization performance with different training data.
['Test set', 'Training set', 'ROUGE-1', 'ROUGE-2']
To verify the effect of TGSum, we examine whether it can be used to improve the performance of summarization systems on DUC benchmarks. To be concrete, each sentence in the training set is scored by ROUGE-2. Then we extract the features such as TF (the averaged TF scores of the sentence), LENGTH (sentence length), and STOP-RATIO (the ratio of stopwords), and train SVR to measure the saliency of sentences. It is a standard learning-based summarization model, which is enough to emphasize the effect of training data. We train this summarizer on different datasets and test it on DUC. Here “DUC” stands for DUC datasets except the testing year, and “TWT_REF” means the same documents as TGSum but directly use tweets as reference summaries. Seeing from this table, ROUGE scores always enjoy a considerable increase when adding TGSum as an extra training set. In addition, even only given the TGSum dataset for training, the summarizer can still achieve comparable performance, especially on DUC 02. Thus, the effect of TGSum references is similar to the manual annotations of DUC, although the former is generated according to tweets automatically. In comparison to TGSum, directly using tweets as references does not always improve the summarization performance. The noise in tweets may misguide the model learning. For example, when treating tweets as references, the feature STOP-RATIO holds extremely high positive weight. This improper feature weight can be ascribed to the informal writing style on Twitter, and it obviously fails to match the informativeness requirement.