Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Table 4: Performance of the three disparate pre-training objectives described in Section 3.3.1.
['Objective', 'GLUE', 'CNNDM', 'SQuAD', 'SGLUE', 'EnDe', 'EnFr', 'EnRo']
Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks. Indeed, the motivation for the BERT objective was to outperform language model-based pre-training. The deshuffling objective performs considerably worse than both prefix language modeling and the BERT-style objective.