Split and Rephrase: Better Evaluation and a Stronger Baseline
Table 5: Results over the test sets of the original, our proposed split and the v1.0 split
['original data split', 'Source', 'BLEU 55.67', '#S/C 1.0', '#T/S 21.11']
On the original data-split, the Copy512 model outperforms all baselines, improving over the previous best by 8.68 BLEU points. On the new data-split, as expected, the performance degrades for all models, as they are required to generalize to sentences not seen during training. The copy-augmented models perform better than the baselines in this case as well, with a larger relative gap which can be explained by the lower lexical overlap between the train and the test sets in the new split. On the v1.0 split the results are similar to those on our split, in spite of it being larger ( 1,331,515 vs. 886,857 examples), indicating that merely adding data will not solve the task.