Bi-Decoder Augmented Network for Neural Machine Translation
Table 1: BLEU scores for NMT models on WMT14 English-German and English-French and IWSLT 2015 English-Vietnamese dataset. “AD” denotes auxiliary decoder, “RL” denotes reinforcement learning.
['[BOLD] NMT Models', '[BOLD] WMT14  [BOLD] En→De', '[BOLD] WMT14  [BOLD] De→En', '[BOLD] WMT14  [BOLD] En→Fr', '[BOLD] IWSLT15  [BOLD] En→Vi']
In the medium part, we conduct an ablation experiment to evaluate the individual contribution of each component of our model. At first, we only add the auxiliary decoder D2 to the baseline model, and the BLEU scores on all the test sets rise about 1.4 point, which shows the effectiveness of our bi-decoder architecture and the significance of the language-independent representation of the text. We then train our model with the process of denoising, in other words, we take out the objective function JRL when optimizing our BiDAN model. We observe that the performance rises around 0.3 point, which proves that focusing on the internal structure of the languages is very helpful to the task. Finally, we use the reinforcement learning with the original loss to train our auxiliary decoder, which means we don’t use the objective function JD. The results show that this leads to about 0.4 point improvement, which indicates that relaxing the grammatical limitation and capturing the keyword information is very useful in our bi-decoder architecture. Finally, instead of first jointly train the parameters of the whole model to the decoder D1 90% convergence and then fix θ2 (parameters of the D2) and train Θ1 using J1 until the model fully converges, we directly training the model until all of its parts converge. The results show that training from a well-trained model may induce the local minimum of the optimization. In other words, we keep λd and λr as 2 and change the ratio of the original objective J2. As we can see, on all of the datasets the model drops sharply and even worse than the baseline model when we set λa as 0, which means we only use the objective functions JD and JRL without the original one. This indicates that totally ignoring the grammatical structure of the input text is not helpful to the task. We also observe that the performance rises with the increase of λa until 5 or 6. Afterwards, the results get worse when we raise the λa, which means the multi-objective learning can improve the performance.