A Closer Look at Data Bias in Neural Extractive Summarization Models
Table 3: Results of cross-dataset PCR(ηp), CCR(ηc) and ROUGE-2 score. Each cell ηpij and ηcij denotes the coverage rate between training dataset (rows) and test dataset (columns). Each cell R-2ij denotes model performance in cross-dataset setting.
['[EMPTY]', 'Dataset', 'CNNDM', 'arXiv', 'Pubmed', 'NYT50', 'Newsr.']
We calculate PCR/CCR score and measure the performance of the base model by ROUGE-2 score on five datasets. Furthermore, model performance is also in accord with this trend. Consistency presented by the experiment, on the one hand, illustrates that there are significant shifts between different datasets, which results in performance differences of the model in cross-dataset setting, on the other hand, it reflects that position distribution and content information are the key factors of such dataset-shift. 1) For in-dataset setting, comparing the Tag and the basic models, we find a very simple method that assign each sample a domain tag could achieve improvement. The reason here we claim is that domain-aware model makes full use of the nature of dataset. 2) For multi-domain and meta-learning model, we attempt to explain from the perspective of data distribution. Although meta-learning obtains worse performance under in-dataset setting, it yet has achieved impressive performance under cross-dataset setting. Correspondingly, Tag model uses a randomly initialized embedding for zero-shot transfer, and we suspect that this perturbation unexpectedly generalizes well on some far-distributed datasets (arXiv and PubMed). 3) BERT has shown its superior performance and nearly outperforms all competitors. However, the generalization ability of BERT is poor on arXiv, PubMed and DUC2002 compared to the performance improvement in in-dataset setting. In contrast, BERT shows good generalization when tranferring to datasets with high density and compression (NYT50 and Newsroom). As we have discussed in Sec.