Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
Table 5: The impact of λgen on MCIC-COCOaccuracy, together with caption-generation performance (ROUGE-L and CIDEr against 5 references). All results are obtained with a Vec2seq+FFNNmodel (embedding size 2048 and hidden-layer sizes of 2048 and 512).
['[ITALIC] λgen', 'Acc Dev', 'Acc Test', 'ROUGE-L Dev', 'ROUGE-L Test', 'CIDEr Dev', 'CIDEr Test']
That is, the ability to perform the comprehension task (as measured by the accuracy metric) positively correlates with the ability to perform other tasks that require machine comprehension, such as caption generation. At λgen=4, the Vec2seq+FFNNmodel not only has a high accuracy of detecting the ground-truth option, but it also generates its own captions given the input image, with an accuracy measured on MCIC-COCOat 0.9890 (dev) and 0.9380 (test) CIDEr scores. On the other hand, at an accuracy level of about 59% (on test, at λgen=0.1), the generation performance is at only 0.9010 (dev) and 0.8650 (test) CIDEr scores.