Parallel Scheduled Sampling
Table 2: Performance on the summarization task using base and large Transformer when trained with teacher-forcing and Parallel Scheduled Sampling. We consider both beam search and greedy decoding. We adopt the widely-used ROUGE score as the evaluation metric (higher the better).
['Model Size', 'Max Length', 'Training Method', 'Decoding Method', 'ROUGE-2', 'ROUGE-L']
The base and large models were trained for 250k steps and 500k steps respectively. We use teacher-forcing for the first 50% of training steps in Parallel Scheduled Sampling as warm-up steps. With the base model, Parallel Scheduled Sampling obtains better performance than teacher-forcing with both beam search and greedy decoding while it performs better only with greedy decoding when the large model is used. Since we use models that are much bigger than the ones in the dialog task discussed before, it is runtime infeasible to apply Sequential Schedule Sampling here.