MacNet: Transferring Knowledge from Machine Comprehension to Sequence-to-Sequence Models
Table 2: BLEU scores on official test sets (WMT English-German for newstest2014 and newstest2015). In the top part, we show the performance of our baseline model; In the medium part, we present the ablation experiments; In the bottom part, we show the effectiveness of our MacNet.
['[BOLD] NMT Systems', '[BOLD] WMT14  [BOLD] En→De', '[BOLD] WMT14  [BOLD] De→En', '[BOLD] WMT15  [BOLD] En→De', '[BOLD] WMT15  [BOLD] De→En']
In the medium part, we conduct an ablation experiment to evaluate the individual contribution of each component of our model. Both of the encoding layer and the modeling layer demonstrates their effectiveness when we ablate other modules. When we add both of them (still without the focal loss), the BLEU scores on all the test sets rise at least 1 point, which shows the significance of the transferred knowledge. Finally, we add the architecture of the encoding layer and the modeling layer to the baseline model but initialize them randomly as its other RNN layers. We observe that the performance drops around 0.5%, which indicates that the machine comprehension knowledge has deep connections with the machine translation tasks. From the ablation experiments we found that the improvement of the modeling layer in our architecture is a bit modest, but we believe transferring high-level networks (e.g. the modeling layer) can help a lot with a more suitable structure because those networks contains deeper semantic knowledge and more abstractive information compared with the lower-level layers (e.g. encoding layer).