Learning Compact Reward for Image Captioning
Table 3: Sentence-level correlation with human evaluation. All p-value (not shown) are less than 0.001.
['Method', 'Correctness Peason', 'Correctness Spearman', 'Correctness Kendall', 'Throughness Peason', 'Throughness Spearman', 'Throughness Kendall']
As a caption evaluator, the discriminator learns gθ that recovers the true reward up to a constant at optimality In the human evaluation process, the AMT worker was asked to give a score at range of 1-5 to evaluate the correctness and throughness of each sentence. Captions with length exceeding 20 were discarded, resulting a total of 11,657 sentences. The correlation is evaluated using Pearson p, Kendall’s τ and Spearman’s r correlation coefficients. Among the reward-learning methods, AIRL poorly correlates with human, whereas the proposed rAIRL improves AIRL on all the correlation metrics, especially on the Pearson correlation (from 0.04 to 0.43). Furthermore, a simple combination of SPICE and the recovered reward leads to an increased correlation with the human scores, which proves the capacity of the discriminator as a caption evaluator. We also found that conventional metrics, especially BLEU, do not correlate well with human evaluation in terms of linear correlation. Therefore, in the experiments of evaluating the captions in the next section, we directly adopt human studies as the evaluation method, along with other objective evaluation metrics that have proven to correlate well with human, including SPICE Results on the conventional metrics are also reported for comparison with existing methods, but they are not addressed.