Does Multimodality Help Human and Machine for Translation and Image Captioning?
Table 6: BLEU and METEOR scores for human description generation experiments.
['Method', 'BLEU-1', 'BLEU-2', 'BLEU-3', 'BLEU-4', 'METEOR']
To evaluate the importance of the different modalities for the image description generation task, we have performed an experiment where we replace the computer algorithm with human participants. The two modalities are the five English description sentences, and the image. The output is a single description sentence in German. The experiment asks the participants for the following tasks: Given both the image and the English descriptions: ’Describe the image in one sentence in German. You can get help from the English sentences provided.’ Given only the image: ’Describe the image in one sentence in German.’ Given only one English sentence: ’Translate the English sentence into German.’ The experiment was performed by 16 native German speakers proficient in English with age ranging from 23 to 54 (coming from Austria, Germany and Switzerland, of which 10 are female and 6 male). The experiment is performed on the first 80 sentences of the validation set. Participants performed 10 repetitions for each task, and not repeating the same image across tasks. For humans, the English description sentences help to obtain better performance. Removing the image altogether and providing only a single English description sentence results in a significant drop. We were surprised to observe such a drop, whereas we expected good translations to obtain competitive results. In addition, we have provided the results of our submission on the same subset of images; humans clearly obtain better performance using METEOR metrics, but our approach is clearly outperforming on the BLEU metrics. The participants were not trained on the train set before performing the tasks, which could be one of the reasons for the difference. Furthermore, given the lower performance of only translating one of the English description sentences on both metrics, it could possibly be caused by existing biases in the data set.