Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
Table 3: Results of different models and initialization setups on WikiSplit. Blockwise sorted by SARI score.
['WikiSplit', 'Exact', 'SARI', 'BLEU']
The reverse task of sentence fusion is the split-and-rephrase task, which requires rewriting a long sentence into two or more coherent short sentences Narayan et al. We use the WikiSplit dataset Botha et al. Training was done for 300k steps with a global batch size of 256. The input and output are padded to a length of 128, which covers 100% of the training, evaluation and test data. Previous state-of-the-art results by \newcitewikisplit used a bi-directional LSTM with a copy mechanism Aharoni and Goldberg The shared encoder-decoder setup of bertShare outperforms all other setups. For the larger models with 24 layers, we observed a small over-fitting after 100k steps (~25 epochs), and therefore stop the training early. bertShare and robertaShare perform on par and both outperform their 12 layer counterpart.