Object Counts! Bringing Explicit Detections Back into Image Captioning
Table 2: CIDEr scores for captioning comparing the use of min, max or average pooling of either object size or distance features, using ground truth annotations.
['Feature vs. Pooling', '[BOLD] Min', '[BOLD] Max', '[BOLD] Mean']
We also experimented with different methods for aggregating multiple instances of the same category, in addition to choosing the biggest instance and the instance closest to the image centre. For example, choosing the smallest instance (min pooling) or the instance furthest away from the image centre (max pooling), or just averaging them (mean pooling). For object size, the findings are as expected: Smaller object instances are less important for IC, although averaging them works comparably well. Surprisingly, in the case of distance, using the object furthest from the image centre actually gave slightly better results than the one closest. Further inspection revealed that aggregating instances is not effective in some cases. We found that the positional information (and interaction with other objects) captured by the object further away may sometimes represent the semantics of the image better than the object in the centre of the image. Representing the person as the one furthest from the image will result in some inference (from training data) that there could be more than one person in the image sitting around the kitchen counter rather than a single person standing at the kitchen counter.