Multi-Agent Cross-Translated Diversification for Unsupervised Machine Translation
Table 6: BLEU comparison of MACD vs. an ensemble of UMT agents and ensemble knowledge distillation (ensemble_distill_freitag2017) on WMT’14 En-Fr, WMT’16 En-De and En-Ro translation tasks.
['[BOLD] Method', '[BOLD] En-Fr', '[BOLD] Fr-En', '[BOLD] En-De', '[BOLD] De-En', '[BOLD] En-Ro', '[BOLD] Ro-En']
Since MACD utilizes outputs from multiple UMT agents to improve the baseline system, it is interesting to see how does it perform compared to a simple ensemble of UMT models and ensemble knowledge distillation (ensemble_distill_freitag2017). To perform ensembling, we average the probabilities of 2 UMT agents at each decoding step. For ensemble distillation, we generate synthetic parallel data from an ensemble of UMT agents, which is then used to train the supervised model. The results show that MACD is capable of leveraging the potentials of multiple UMT agents better than how an ensemble of agents does. This is in contrast to data diversification (nguyen2019data), which is shown to mimic and perform similarly to model ensembling. More importantly, during inference, an ensemble of models requires more memory and computations to store and execute multiple models while MACD needs only one model. Meanwhile, ensemble knowledge distillation (ensemble_distill_freitag2017), which performs well with supervised agents, performs poorly in unsupervised tasks, perhaps because the UMT agents are not capable enough for the method intended for supervised learning. Further inspection in the Appendix suggests that many samples in the ensemble translations contain incomprehensible repetitions.