Unsupervised Neural Machine Translation with Weight Sharing
Table 3: Ablation study on English-German, English-French and Chinese-to-English translation tasks. Without weight sharing means no layers are shared in the two AEs.
['[EMPTY]', 'en-de', 'de-en', 'en-fr', 'fr-en', 'zh-en']
, we perform an ablation study by training multiple versions of our model with some missing components: the local GANs, the global GANs, the directional self-attention, the weight-sharing, the embedding-reinforced encoders, etc. We do not test the the importance of the auto-encoding, back-translation and the pre-trained embeddings because they have been widely tested in Lample et al. The most critical component is the weight-sharing constraint, which is vital to map sentences of different languages to the shared-latent space. The embedding-reinforced encoder also brings some improvement on all of the translation tasks. When we remove the directional self-attention, we get up to -0.3 BLEU points decline. This indicates that it deserves more efforts to investigate the temporal order information in self-attention mechanism. The GANs also significantly improve the translation performance of our system. Specifically, the global GANs achieve improvement up to +0.78 BLEU points on English-to-French translation and the local GANs also obtain improvement up to +0.57 BLEU points on English-to-French translation. This reveals that the proposed model benefits a lot from the cross-domain loss defined by GANs.