Pre-training via Paraphrasing
Table 5: ROUGE-L scores on MLSum. MARGE generates abstractive summaries that outperform an extractive mBERT model. We also demonstrate zero-shot transfer learning, where the model is trained only on languages it is not trained on, and results from training on all languages.
['Model', 'MLSum Setting', 'MLSum de', 'MLSum es', 'MLSum fr', 'MLSum ru', 'MLSum tr', 'avg']
In some cases, training one model on all languages (train all) improves results. Finally, we explore zero-shot summarization, where the model is trained on all languages except the test language—this model outperforms a strong lead-3 baseline, and even a supervised pointer-generator model on Spanish and Russian. On this domain, we achieve better results with MARGE-NEWS, a version of the model trained only on news.