Hierarchy Parsing for Image Captioning
Table 1: Performance (%) of our HIP and other methods on COCO Karpathy test split.
['[EMPTY]', '[BOLD] Cross-Entropy Loss', '[BOLD] Cross-Entropy Loss', '[BOLD] Cross-Entropy Loss', '[BOLD] Cross-Entropy Loss', '[BOLD] Cross-Entropy Loss', '[BOLD] CIDEr-D Score Optimization', '[BOLD] CIDEr-D Score Optimization', '[BOLD] CIDEr-D Score Optimization', '[BOLD] CIDEr-D Score Optimization', '[BOLD] CIDEr-D Score Optimization']
Performance on COCO. Overall, the results across all metrics and two optimization methods (Cross-Entropy Loss and CIDEr-D Score Optimization) consistently indicate that our GCN-LSTM+HIP exhibits better performances than other approaches, including non-attention models (LSTM, LSTM-A) and attention-based approaches (SCST, ADP-ATT, RFNet, Up-Down, and GCN-LSTM). Up -Down+HIP and GCN-LSTM+HIP by integrating hierarchy parsing architecture makes the absolute improvement over Up-Down and GCN-LSTM by 3.1% and 3.2% in terms of CIDEr-D respectively, optimized with cross-entropy loss. The results generally highlight the key advantage of exploiting the hierarchal structure in an image from instance level, region level, to the whole image, pursuing a thorough image understanding for captioning. Specifically, by injecting the high-level semantic attributes into LSTM-based decoder, LSTM-A outperforms LSTM that trains decoder only depending on the input image. Nevertheless, the attention-based methods (SCST, ADP-ATT, Up-Down, and RFNet) exhibit better performance than LSTM-A, which verifies the merit of attention mechanism that dynamically focuses on image regions for sentence generation. Furthermore, GCN-LSTM by exploring the relations between objects to enrich region-level features, improves SCST, ADP-ATT, Up-Down, and RFNet. However, the performances of GCN-LSTM are lower than GCN-LSTM+HIP that additionally exploits hierarchical structure in an image for enhancing all the instance-level, region-level and image-level features and eventually boosting image captioning. In addition, by optimizing the captioning models with CIDEr-D score instead of cross-entropy loss, the CIDEr-D score of GCN-LSTM+HIP is further boosted up to 130.6%. This confirms that self-critical training strategy is an effective way to amend the discrepancy between training and inference, and improve sentence generation regardless of image captioning approaches. Similar to the observations on the optimization with cross-entropy loss, Up-Down+HIP and GCN-LSTM+HIP lead to better performances than Up-Down and GCN-LSTM when optimized with CIDEr-D score.