Object Counts! Bringing Explicit Detections Back into Image Captioning
Table 3: k-Nearest Neighbour (k=5) trial on the ground truth bag of objects (Freq.) and the projected bag of objects (Proj.) representations. The references are captions of 5-nearest images in each space. Exact represents a subset of 2301 samples where all the 5 neighbours have 0 distance (replicas) and ¬ represents nearest neighbours that are not replicas of the test image.
['Type', 'BLEU', 'Meteor', 'CIDEr', 'SPICE']
Further analysis showed that 2301 out of 5000 captions had nearest images at a zero distance, i.e., the same exact representation was seen at least 5 times in training (note that CIDEr gives a score of 10 only if the test caption and all references are the same). We found that among the non-exact image matches, the projected image representation better captures candidates in the training set than bag of objects. Note that the nearest neighbours are an approximation since we do not know the exact distance metric derived from the LSTM. We observe that the captions for unseen representations seem to be interpolated from multiple neighbouring points in the projection space, but further work is needed to analyze the hidden representations of the LSTM to understand the language model and to give firmer conclusions.