Sample Efficient Text Summarization Using a Single Pre-Trained Transformer
Table 1: Summarization results when using the full training set. Our scores are averaged over three models trained with different random seeds. *Other abstractive summarization model scores are provided to contextualize performance on this task but are not directly comparable to our models.
['[BOLD] Model', '[BOLD] R1', '[BOLD] R2', '[BOLD] RL']
We find that pre-training improves performance by about 2 ROUGE points, on average. Surprisingly, when only the decoder is pre-trained, ROUGE gets substantially worse. We speculate this is because the model starting out with a well-trained decoder and poor encoder learns to overly rely on its language modeling abilities and not adequately incorporate information from the encoder. The Transformer LM outperforms corresponding models both with and without pre-training, despite having almost half as many parameters. Our best model performs competitively with existing models on the CNN/Daily Mail abstractive summarization task, despite the absence of model augmentations such as a copy mechanism and reinforcement learning objectives.