Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Table 2: Results of different models and initialization techniques on DiscoFuse and subsampled training sets. Blockwise sorted by SARI score on 100% of the training set.
['DiscoFuse', '100% Exact', '100% SARI', '10% SARI', '1% SARI']
Training was done for 300k steps with a global batch size of 256. The input and output are padded to a length of 128, which covers 100% of the training, evaluation and test data. We report SARI Xu et al. Previous state-of-the-art results by \newcitediscofuse used the vanilla transformer model by \newcitetransformer, with only 7 layers. All models with initialized encoders outperform the baseline by a large margin, with a SARI score of 89.3 compared to 86.9 (bert2rnd vs. rnd2rnd). To measure the effect on smaller training sets, we randomly subsample the training data down to 10% and 1%, i.e. 450k and 45k training examples, respectively. First, we notice, that performance comparable to the baseline is achieved even when training on only 10% of the training data (rnd2rnd vs. robertaShare). Secondly, when using only 1% of the training data setups with fewer randomly initialized parameters (bert2bert vs. bert2rnd) perform better. The best performing 12 layer setup is roberta2gpt with a SARI score of 89.9 only outperformed by 24 layer setup of robertaShare with a SARI score of 90.3.