A Visual Attention Grounding Neural Modelfor Multimodal Machine Translation
Table 3: Translation results on the IKEA dataset
['Method', 'English → German BLEU', 'English → German METEOR', 'English → French BLEU', 'English → French METEOR']
We then evaluate all models on the IKEA dataset. Our VAG-NMT has a higher value in BLEU and a comparable value in METEOR compared to the Text-only NMT baseline. Our VAG-NMT outperforms LIUMCVC’s multimodal system by a large margin, which shows that the LIUMCVC’s multimodal’s good performance on Multi30K does not generalize to this real-world product dataset. The good performance may come from the visual attention mechanism that learns to focus on text segments that are related to the images. Such attention therefore teaches the decoder to apply the visual context to translate those words. This learned attention is especially useful for datasets with long sentences that have irrelevant text information with respect to the image.