COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval
TABLE V: Human evaluation of different models for image tagging on COCO-CN test100. Cascading MLP again performs the best.
['[BOLD] Model', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F-measure']
Human evaluation. It is possible that the lower performance of Clarifai is caused by a discrepancy between the Chinese vocabulary of the online service and our ground-truth vocabulary. To resolve this uncertainty, we performed a user study as follows. For each of the 100 pre-specified images, we collected the top 5 predicted tags by each model. Eight subjects participated in the user study. Each image together with the collected tags was shown to two subjects, who independently rated each of the tags as relevant, irrelevant or unsure. To avoid bias, the tags were randomly shuffled in advanced. Only tags rated as relevant by both subjects were preserved. As the ground truth is more complete, all the scores improve. Cascading MLP again performs the best. Moreover, the qualitative conclusion concerning which model performs better remains the same.