Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Table 6: SARI scores on the Discofuse dataset when experimenting with different embedding setups. Each row also includes the setups of all previous rows.
['[EMPTY]', 'bertShare', 'gpt']
S5SS0SSS0Px4 Initializing only Layers. Contrary to the previous paragraph, we want to investigate the effect of initializing everything but the word embedding matrix. The embedding matrix accounts for only 10-31% of all learnable parameters and sometimes the vocabulary given from a public checkpoint might not be optimal for a certain task. In these cases, it would be nice to redefine the vocabulary while still leveraging the checkpoint. The latter is probably due to the large vocab of the GPT-2 model which now remains random initialized. We observe almost no change on bertShare, suggesting that the BERT vocabulary was already optimal for DiscoFuse. gpt however, showed a significant improvement using this much smaller vocabulary but is still behind the fully initialized setup. Finally, we experimented with a more sensitive way of training the model, meaning that we fix all warm-started variables for 100k steps. During this pre-training phase, we only train the new word embeddings. After the pre-training, we fine-tune the entire model for another 300k steps. This training scheme resulted in an improvement of 0.5 for the bertShare setup, but overall the number is still way behind the fully initialized setup. For gpt, this training scheme did not result in a satisfying training curve.