Towards Content Transfer through Grounded Text Generation
Table 2: Automated metrics; 95% confidence interval in parentheses.
['Model SB', 'ROUGE-L 5.6', 'ROUGE-L (5.6–5.7)', 'BLEU 0.6', 'METEOR 2.0']
Furthermore, the results emphasize the importance of having curated text as context when generating the update. In all experimental conditions, models aware of context perform much better than models agnostic of it. In contrast to Liu et al. Extractive CAG, CIG, and CRG all outperformed both Sum-Basic and the context informed variant. Extractive CAG was on-par with generative CAG, suggesting the generated sentences were of reasonable quality. However, generative CIG and CRG were substantially better: rewriting to match context was beneficial. Note this yields a very optimistic upper bound, as the same ground truth xi is used both to select an extractive sentence from a large pool of candidates and for final automatic metric scoring. Nevertheless, these oracle results let us draw two conclusions: (1) They give us better perspective to assess the non-oracle systems, and we believe that their seemingly low automatic evaluation scores are quite reasonable relative to the optimistic upper bound (e.g., CIG’s ROUGE-L’s score is 55% of the oracle). (2) The oracle results suggest that humans are substantially changing the surface realization as they summarize for Wikipedia, as otherwise the oracle results would be much closer to maximum metric scores (i.e., 100%). This shows that extractive methods are not enough for this task, justifying our use of generation techniques.