Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Table 1: Average and standard deviation of scores achieved by our baseline model and training procedure. For comparison, we also report performance when training on each task from scratch (i.e. without any pre-training) for the same number of steps used to fine-tune the baseline model. All scores in this table (and every table in our paper except Table 14) are reported on the validation sets of each dataset.
['[EMPTY]', 'GLUE', 'CNNDM', 'SQuAD', 'SGLUE', 'EnDe', 'EnFr', 'EnRo']
Our results tables are all formatted so that each row corresponds to a particular experimental configuration with columns giving the scores for each benchmark. We will include the mean performance of the baseline configuration in most tables. We also will boldface any score that is within two standard deviations of the maximum (best) in a given experiment. Overall, our results are comparable to existing models of similar size. Note that we cannot directly compare our baseline to BERTBASE because ours is an encoder-decoder model and was pre-trained for roughly \sfrac14 as many steps. Unsurprisingly, we find that pre-training provides significant gains across almost all benchmarks. The only exception is WMT English to French, which is a large enough dataset that gains from pre-training tend to be marginal. We include this task in our experiments to test the behavior of transfer learning in the high-resource regime.