Does Multimodality Help Human and Machine for Translation and Image Captioning?
Table 5: BLEU and METEOR scores of our NMT based submissions for Task 2.
['System', 'Validation METEOR', 'Validation BLEU', 'Test METEOR', 'Test BLEU']
Several explanations can clarify this behavior. First, the architecture is not well suited for integrating image and text representations. This is possible as we did not explore all the possibilities to benefit from both modalities. Another explanation is that the image context contain too much irrelevant information which cannot be discriminated by the lone attention mechanism. This would need a deeper analysis of the attention weights in order to be answered.