Visual Commonsense Graphs: Reasoning about the Dynamic Context of a Still Image visualcomet.xyz
Table 2: Ablation Results. Ablations of our baseline model on the Validation set. We use nucleus sampling with p=0.9 to generate 5 sentences for all models. We report automatic evaluations on quality, coherence, and diversity of generated inferences. EP loss indicates if the model is trained to predict event at present and place. Text Given is when model is given any textual input during test time to generate the inferences. We bold the models based on the following order: 1) Best text only model, 2) Best image and text only model given visual and text input, 3) Best image only model, and 4) Best image and text model given just visual input.
['[BOLD] Modalities', '[BOLD] Text ', '[BOLD] BLEU-2 \\uparrow', '[BOLD] Acc@50 \\uparrow', '[BOLD] Unique', '[BOLD] Novel']
We make the following observations: 1) Adding PG trick gives a boost for model with image modalities. 2) Model trained with both vision and text (Image + Event + Place + PG) modalities outperform models trained with only one of modality (Event + Place; Image + PG) in every metric, including retrieval accuracy and diversity scores. This indicates that the task needs visual information to get higher quality inferences. 3) Adding place information helps in general. 4) Models with access to textual event and place information during test time, generate higher quality sentences than the same models without them ( Text Given Yes vs No). This is not surprising as our dataset was collected with workers looking at the event, and the event already gives a strong signal understanding the content in the image. Interestingly, the Acc@50 gets a higher score when textual input is not given, even for the model trained with the visual and textual input (Image + Event + Place). This suggests that the image and text model switches to be more visually coherent when textual signal is not given in the test time. 5) Lastly, adding the EP Loss boosts the performance if only the image content is available in the test time. This indicates when the model is forced to generate inferences directly from image, having it go through the two-stage pipeline of recognizing events and then making inferences, helps the generation performance. While the numbers in automatic evaluation give favorable results to our Image + Text model, they are not sufficient enough to evaluate the quality of generated inferences. We take 200 random images and the generated inferences, and ask the humans to evaluate their quality based on just the image content. Even for models that use ground truth inferences, we do not show the events to the workers and make them rely on image to make the decision. Specifically, we ask three different workers to evaluate if each inference is likely (1) or unlikely (0) to happen based on the image. We then take the majority out of three and calculate the average across all the inferences.