Expressing Visual Relationships via Language
Table 3: Human evaluation on 100 examples. Image pair and two captions generated by our basic model and full model are shown to the user. The user chooses one from ‘Basic’ model wins, ‘Full’ model wins, ‘Both Good’, or ‘Both Not’. Better model marked in bold font.
['[EMPTY]', 'Basic', 'Full', 'Both Good', 'Both Not']
We conduct a pairwise human evaluation on our generated sentences, which is used in \newcitecelikyilmaz2018deep and \newcitepasunuru2017reinforced. \newciteagarwala_2018 also shows that the pairwise comparison is better than scoring sentences individually. We randomly select 100 examples from the test set in each dataset and generate captions via our full speaker model. We ask users to choose a better instruction between the captions generated by our full model and the basic model, or alternatively indicate that the two captions are equal in quality. The Image Editing Request dataset is specifically annotated by the image editing expert. Our full model outperforms the basic model significantly. We also show positive and negative examples generated by our full model in Fig. In our Image Editing Request corpus, the model was able to detect and describe the editing actions but it failed in handling the arbitrary complex editing actions. We keep these hard examples in our dataset to match real-world requirements and allow follow-up future works to pursue the remaining challenges in this task. Our model is designed for non-localized relationships thus we do not explicitly model the pixel-level differences; however, we still find that the model could learn these differences in the Spot-the-Diff dataset. Since the descriptions in Spot-the-Diff is relatively simple, the errors mostly come from wrong entities or undetected differences as shown in Fig. Our model is also sensitive to the image contents as shown in the NLVR2 dataset.