Token-level and sequence-level loss smoothing for RNN language models
Table 3: Tokenized BLEU score on WMT’14 En-Fr evaluated on the news-test-2014 set. And Tokenzied, case-insensitive BLEU on IWSLT’14 De-En.
['Loss', 'Reward', 'V [ITALIC] sub', '[BOLD] WMT’14', '[BOLD] IWSLT’14']
On both benchmarks, we improve on both MLE and RAML approach of Norouzi et al. (Seq, Hamming, V): using the smaller batch-vocabulary for replacement improves results, and using importance sampling based on BLEU-4 further boosts results. In this case, unlike in the captioning experiment, token-level smoothing brings smaller improvements. The combination of both smoothing approaches gives best results, similar to what was observed for image captioning, improving the MLE BLEU-4 from 30.03 to 31.39 on WMT’14 and from 27.55 to 28.74 on IWSLT’14.