Imagination improves Multimodal Translation
Table 2: En→De translation results on the Multi30K dataset. Our Imagination model is competitive with the state of the art when it is trained on in-domain data. We report the mean and standard deviation of three random initialisations.
['[EMPTY]', 'Meteor', 'BLEU']
We can see that the combination of the attention-based translation model and the image prediction model is a 1.8 Meteor point improvement over the NMT baseline, but it is 1.1 Meteor points worse than the strong Moses baseline. Our approach is competitive with previous approaches that use visual features as inputs to the decoder and the target-side reranking model. It also competitive with \newcitetoyama2016neural, which also only uses images for training. These results confirm that our multitasking approach uses the image prediction task to improve the encoder of the translation model.