Object Counts! Bringing Explicit Detections Back into Image Captioning
Table 4: CIDEr scores for image captioning using representations encoding spatial information of instances derived from ground truth annotations, with either fixed hyperparameters (Section 3.1) or with hyperparameter tuning. † Results taken from Yin and Ordonez (2017).
['Feature set', 'Fixed', 'Tuned']
All experiments in this subsection use ground truth annotations – we expect the results of using an object detector to be slightly worse but in most cases follow a similar trend, as shown in the previous section. Encoding spatial information led to substantially better performance over bag of objects alone. Consistent with our previous observation, w and h (bounding box width and height) seems to be the most informative feature combination – it performs well even without positional information. Area (a) is less informative than the combination of w and h, possibly because it compresses width-height ratio information despite discarding noise from background regions. Positional information (x, y) does not seem to be as informative, consistent with observations from previous work We note that the results with our simpler image representation are comparable to the ones reported in \newciteYinOrdonez:2017, which use more complex models to encode similar image information. Interestingly, we observe that positional information (x, y) work better than before tuning in this case.