Classical Structured Prediction Losses for Sequence to Sequence Learning
Table 3: Effect of initializing sequence-level training (Risk) with parameters from token-level likelihood (TokNLL) or label smoothing (TokLS).
['[EMPTY]', '[BOLD] valid', '[BOLD] test']
So far we initialized sequence-level models with parameters from a token-level model trained with label smoothing. The improvement of initializing with TokNLL is only 0.3 BLEU with respect to the TokNLL baseline, whereas, the improvement from initializing with TokLS is 0.6-0.8 BLEU. We believe that the regularization provided by label smoothing leads to models with less sharp distributions that are a better starting point for sequence-level training.