Captioning Images with Diverse Objects
Table 6: ImageNet: Human judgements comparing our NOC model with DCC [7] on the ability to meaningfully incorporate the novel object in the description (Word Incorporation) and describe the image. ‘Union’ and ‘Intersection’ refer to the subset of objects where atleast one model, and both models are able to incorporate the object name in the description. All values in %.
['Objects subset →', 'Word Incorporation Union', 'Word Incorporation Intersection', 'Image Description Union', 'Image Description Intersection']
ImageNet images do not have accompanying captions and this makes the task much more challenging to evaluate. To compare the performance of NOC and DCC we obtain human judgements on captions generated by the models on several object categories. We select 3 images each from about 580 object categories that at least one of the two models, NOC and DCC, can describe. (Note that although both models were trained on the same ImageNet object categories, NOC is able to describe almost all of the object categories that have been described by DCC). When selecting the images, for object categories that both models can describe, we make sure to select at least two images for which both models mention the object label in the description. Each image is presented to three workers. We conducted two human studies (sample interface is in the supplement): Given the image, the ground-truth object category (and meaning), and the captions generated by the models, we evaluate on: Word Incorporation: We ask humans to choose which sentence/caption incorporates the object label meaningfully in the description. The options provided are: (i) Sentence 1 incorporates the word better, ( ii) Sentence 2 incorporates the word better, (iii) Both sentences incorporate the word equally well, or (iv) Neither of them do well. Image Description: We also ask humans to pick which of the two sentences describes the image better. This allows us to compare both how well a model incorporates the novel object label in the sentence, as well as how appropriate the description is to the image. On the subset of images corresponding to objects that both models can describe (Intersection), NOC and DCC appear evenly matched, with NOC only having a slight edge. However, looking at all object categories (Union), NOC is able to both incorporate the object label in the sentence, and describe the image better than DCC.