Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Table 5: Summarization results of different models and their initialization setups. We compare our setups (the bottom block) against both non-pre-trained (the top block) and pre-trained (the middle block) models on various datasets: the Lead baseline, PtGen See et al. (2017), ConvS2S Gehring et al. (2017), MMN Kim et al. (2019), Bottom-Up Gehrmann et al. (2018), MASS Song et al. (2019), TransLM Khandelwal et al. (2019) and UniLM Dong et al. (2019). The Lead results for the CNN/DailyMail dataset is taken from narayan-etal-2018-ranking, whereas, Lead, PtGen and ConvS2S results on the BBC dataset are taken from narayan-etal-2018-xsum. Our best results are boldfaced and the best results on the datasets are italicized.
['[EMPTY]', 'Gigaword R-1', 'Gigaword R-2', 'Gigaword R-L', 'CNN/Dailymail R-1', 'CNN/Dailymail R-2', 'CNN/Dailymail R-L', 'BBC XSum R-1', 'BBC XSum R-2', 'BBC XSum R-L']
In all three cases, we did not anonymize entities. We worked on the original cased versions of CNN/DailyMail and BBC datasets. For Gigaword we used the lowercased version to match the requirements of the publicly available lowercased test set. During training, the input documents were truncated to 512 tokens for CNN/DailyMail and BBC, and to 128 tokens for Gigaword. Similarly, the length of the summaries was limited to 128 tokens for CNN/DailyMail, 64 for BBC, and 32 for Gigaword. We used a global batch size of 128 document-summary pairs for CNN/DailyMail and BBC, and 256 document-summary pairs for Gigaword. We adapted to different number of training steps depending on the training data sizes. Models were trained for 500k, 300k and 200k steps for the Gigaword, CNN/DailyMail and BBC summarization datasets respectively. In all cases, we used the standard publicly available test sets; these consists of 1951 instances for Gigaword, 11490 for CNN/DailyMail and 11334 for BBC. We report on the ROUGE F1 scores Lin and Hovy Pre-training for sequence generation. Pre-training for seq2seq learning was first done by \newciteDBLP:journals/corr/RamachandranLL16. They used a language model to pre-train the encoder and decoder of an RNN seq2seq model. Their method improved BLEU scores on newstest2014 by 3 points and ROUGE-L on CNN/Dailymail also by 3 points. However their BLEU score of 24.7 on newstest2014 En→De, compared to 30.6 in this work, and 29.4 ROUGE-L on CNN/Dailymail, compared to 36.33 also show the superiority of the transformer model as well as the masked language model objective of BERT. MASS Song et al. One advantage of this method is that, in contrast to our setups (except for gpt), the encoder-decoder attention mechanism is also pre-trained. The downside of this approach is that the pre-trained model is task-specific and not as general as BERT or GPT-2. UniLM Dong et al. also unifies bidirectional, unidirectional, and sequence to sequence language modeling. At the time of writing, no public checkpoint was available to us. To overcome the issue that the encoder-decoder attention is not pre-trained, \newciteDBLP:journals/corr/abs-1905-08836 pre-trained a single transformer language model that encodes the source and generates the target. This setup matches our gpt setup. \newciteDBLP: journals/corr/abs-1901-07291 pre-train their model using casual language modeling (like GPT), masked language modeling (like BERT) and a third new objective called translation language modeling to improve cross-lingual pre-training.