End-to-End Learning Using Cycle Consistency for Image-to-Caption Transformations
Table 2: Evaluation of faithfulness using crowdsourcing. Workers chose our methods more than comparison method.
['[EMPTY]', 'faithfulness of generated captions']
To determine whether faithful captions were generated, a manual evaluation was also performed by crowdsourcing using AMT. Given generated caption pairs (from the proposed and comparison methods) together with an image from the dataset, workers were asked to select which caption better represents the image. Here, we instructed the workers to give priority to judging the extent to which the caption refers to the image, rather than focusing on grammatical order. The data employed here consisted of 100 randomly selected items from the test data, and five workers completed the same task. The workers were all certified as Master by AMT. The points obtained by summing the selections of each option for the proposed and comparison methods were used for the evaluation. In addition, binomial tests were performed on the evaluation results, to determine whether the differences between the proposed and comparison methods were significant. Here, it was tested whether the proportion selecting the proposed method was significantly different from 0.5, with a significance level of 0.05. For both datasets, the proposed method achieved higher points, and the statistical tests show a significant difference between the proposed and comparison methods. (Each p value was less than 0.05.)