PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
Table 3: Results of abstractive summarization on the Gigaword test set. Results of DAE (Denoising Auto-Encoder) and BERT+LM are taken from Song et al. (2019). a Klein et al. (2017); b Lin et al. (2018); c Suzuki and Nagata (2017); d Cao et al. (2018); e Cao et al. (2018); f Wang et al. (2019); g Amplayo et al. (2018); h Dong et al. (2019)
['[BOLD] Method', '[BOLD] Rouge-1', '[BOLD] Rouge-2', '[BOLD] Rouge-L']
It is worth noting that UniLM, MASS, BERT+LM and DAE are pre-trained on an unlabeled corpus before supervised fine-tuning on the summarization data. By consistently outperforming these pre-training methods, PALM confirms its effectiveness in leveraging unsupervision signals for language generation.