Show, Recall, and Tell: Image Captioning with Recall Mechanism
Table 1: Experiment of our proposed recall mechanism on the MSCOCO Karpathy test split with both cross-entropy loss and CIDEr optimization. We implement our proposed methods: semantic guide (SG), recalled-word slot (RWS) and recalled-word reward (WR) on the baseline model Up-Down. Test results show that our proposed methods have obvious improvement over our baseline. B-1 / B-4 / M / R / C / S refers to BLEU1/ BLEU4 / METEOR / ROUGE-L / CIDEr / SPICE scores.
['Models', 'Cross-entropy loss B-1', 'Cross-entropy loss B-4', 'Cross-entropy loss M', 'Cross-entropy loss R', 'Cross-entropy loss C', 'Cross-entropy loss S', 'CIDEr optimization training B-1', 'CIDEr optimization training B-4', 'CIDEr optimization training M', 'CIDEr optimization training R', 'CIDEr optimization training C', 'CIDEr optimization training S']
, we have proposed semantic guide (SG), recalled-word slot (RWS) in captioning model and recalled-word reward (WR) in CIDEr optimization. Then we test their performances on MSCOCO Karpathy test split. Beam search with beam size 2 is employed to generate captions. Semantic guide with recalled-word slot (SG+RWS) has improved 3.0% on cross-entropy loss, and 6.8% on CIDEr optimization. Our best model (SG+RWS+WR) has obtained 7.5% improvement on CIDEr optimization, and it has obtained BLEU4 / CIDEr / SPICE scores of 36.6 / 116.9 / 21.3 with cross-entropy loss and 38.5 / 129.1/ 22.4 with CIDEr optimization. In addition, compared with other state-of-the-art models, like SCST [Rennie_2017_CVPR], StacKCap [gu2018stack], and CAVP [liu2018context] , our results still outperform theirs. The results of comparison demonstrate the effectiveness of our proposed methods, especially on those more convincing evaluation metrics such as CIDEr, SPICE.