Visual Agreement Regularized Training for Multi-Modal Machine Translation
Table 3: The results of incremental analysis. The best performance is highlighted in bold and “MHCA” means our proposed multi-head co-attention.
['Methods', 'EN→DE BLEU', 'EN→DE METEOR', 'EN→FR BLEU', 'EN→FR METEOR']
We treat the text-only Seq2Seq model as the base model and cumulatively add each component until the full model is rebuilt. The results show that the multi-head co-attention model can significantly improve the model performance, illustrating that building bidirectional interactions between textual and visual features can produce better co-dependent representations. By constraining bidirectional models to focus on the same region of the image when generating semantically equivalent visual words, the model can make better use of visual information, resulting in more accurate machine translation hypotheses.