TextCaps: a Dataset for Image Captioningwith Reading Comprehension
Table 0.C.1: Comparison of automatic metrics on human captions between TextCaps test and validation set, using 5 human captions per image (evaluating 1 human caption over the remaining 4 and averaging over the 5 runs).
['#', 'Method', 'B-4', 'M', 'R', 'S', 'C']
Third, we evaluate and compare the automatic metrics on human-written captions between the test and validation sets. Since there are only 5 human captions (instead of 6) collected on the validation set , we perform a similar leave-one-out evaluation as mentioned in Sec. 4.1 but using only 5 human captions per image (evaluating 1 human caption over the remaining 4 and averaging over the 5 runs). Pt0A3T1, where the BLEU-4, METEOR, ROUGE_L, and CIDEr metrics are higher on the test set than on the validation set. This is a bit surprising, but also indicates, that there is slight domain shift between validation and test set, which humans are not affected by, rather than that the test set is more difficult in itself as the models’ performance drop might suggest.