Modeling Fluency and Faithfulness for Diverse Neural Machine Translation
Table 1: BLEU scores on three translation tasks. * and ** mean the improvements over Transformer is statistically significant [collins2005clause] (\rho<0.01 and \rho<0.05, respectively).
['[EMPTY]', '[BOLD] CN\\rightarrowEN  [BOLD] MT03', '[BOLD] CN\\rightarrowEN MT04', '[BOLD] CN\\rightarrowEN MT05', '[BOLD] CN\\rightarrowEN MT06', '[BOLD] CN\\rightarrowEN MT08', '[BOLD] CN\\rightarrowEN AVE', '[BOLD] CN\\rightarrowEN \\small{\\Delta}', '[BOLD] EN\\rightarrowDE WMT14', '[BOLD] EN\\rightarrowDE \\Delta', '[BOLD] EN\\rightarrowRO WMT16', '[BOLD] EN\\rightarrowRO \\Delta']
We can see that the improvement brought by +RL and +BOW methods are not great. For the +RL method, as we all know, the training is not stable and thus it is difficult to converge to good optima. For the +BOW, the reason may be it does not model the word order in the training loss and hence has a loose supervision to the fluency. On the CN\rightarrowEN and EN\rightarrowRO translation tasks, the loss \mathcal{L}_{\mathrm{c}} is much more effective than \mathcal{L}_{\mathrm{KL}} while on the EN\rightarrowDE translation the result is reversed. Another finding is that all the methods cannot achieve big improvements on the EN\rightarrowDE translation. The reason may be the training data is big enough and the two languages EN and DE are closed to each other. As a result, Transformer has already got a good enough optimization and it is difficult for further improvements. Our method with \mathcal{L}_{\mathrm{c}} as an additional loss can outperform all the baselines significantly on the CN\rightarrowEN and EN\rightarrowRO translation tasks. Therefore, we can conclude that the evaluation module can help improve translation performance and the loss \mathcal{L}_{\mathrm{c}} is more reasonable.