Joint Image Captioning and Question Answering
Table 5: Comparison of the image captioning task on the MSCOCO Karpathy test split. Our captioner obtains competitive results in automated evaluation metric and more informative compations in terms of relative improvements on VQA task.
['[EMPTY]', 'cross entropy loss BLEU-1', 'cross entropy loss BLEU-4', 'cross entropy loss METEOR', 'cross entropy loss ROUGE-L', 'cross entropy loss CIDEr', 'cross entropy loss SPICE', 'Info.']
In Table. et al. When determine the captions for the images, we picked the most likely question-related captions based on the problem LABEL: pro:caption_selection. Our system, though only select a fraction of annotated caption as supervision, is capable of achieving similar standard scores. However, in term of the informativeness, we observe a large improvement on relative VQA accuracies from 2.22% to 4.11%. The improvements indicate that importance and necessity of adopting question features as inputs to the image captioning sub-system and more question-specific caption as supervision.