Visual Commonsense Graphs: Reasoning about the Dynamic Context of a Still Image visualcomet.xyz
Table 3: Generated Inference Results. BLEU2 [bleu] and Human score for the generated inferences on the Test split. We select 200 random images and generate 5 sentences for each of the three inference type (3000 sentences total). Then, we assign three annotators to determine if each inference sentence is correct, and take the majority vote. The models are chosen based on their best performance when text only, image only, or image & text modalities are available. We also compare the performance when event and place information is given or not. Note that ground truth have a variable number of inference sentences, in fact, less than 5 in most cases.
['[BOLD] Modalities ', '[BOLD] BLEU-2', '[BOLD] Human  [BOLD] Before ', '[BOLD] Human   [BOLD] Intent ', '[BOLD] Human   [BOLD] After ', '[BOLD] Human   [BOLD] Avg ']
We notice a similar pattern based on our automatic metric results. Image + Text models outperform Image only and Text only model, whether the events are given or not. The difference is even more clear between vision + language and language only model (61.0 vs 50.1 on average). Models with access to textual input do perform better, even if the humans do not see the event and place descriptions. This is not surprising as the textual input already describes the relevant details in the image and has a strong signal. Note that there is still a 20 point gap between our best model and ground truth inferences, meaning there is more room to improve our best model.