Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Table 9: Measuring the effect of artificially shrinking our C4 dataset. This results in the dataset being repeated over the course of pre-training, which may result in memorization (see Figure 6).
['Number of tokens', 'Repeats', 'GLUE', 'CNNDM', 'SQuAD', 'SGLUE', 'EnDe', 'EnFr', 'EnRo']
As expected, performance degrades as the dataset size shrinks. We suspect this may be due to the fact that the model begins to memorize the pre-training dataset. Indeed, the model attains significantly smaller training losses as the size of the pre-training dataset shrinks, suggesting possible memorization.