Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Table 8: Results of m-RNN-shared model after applying consensus reranking using nearest neighbors as references (m-RNN-shared-NNref), compared with those of the original m-RNN model on our validation set and MS COCO test server.
['MS COCO val for consensus reranking', 'MS COCO val for consensus reranking B1', 'MS COCO val for consensus reranking B2', 'MS COCO val for consensus reranking B3', 'MS COCO val for consensus reranking B4', 'MS COCO val for consensus reranking CIDEr', 'MS COCO val for consensus reranking ROUGE_L', 'MS COCO val for consensus reranking METEOR']
For BLEU-based consensus reranking, we get an improvement of 3.5 points on our validation set and 3.3 points on the MS COCO test 2014 set in terms of BLEU4 score. For the CIDEr-based consensus reranking, we get an improvement of 9.4 points on our validation set and 9.8 points on the MS COCO test 2014 set in terms of CIDEr. We also show the oracle performance of the ten hypotheses, which is the upper bound of the consensus reranking. More specifically, for each image in our validation set, we rerank the hypotheses according to the scores (BLEU or CIDEr) w.r.t to the groundtruth captions. (see rows with “-oracle”). The oracle performance is surprisingly high, indicating that there is still room for improvement, both for the m-RNN model itself and the reranking strategy.