Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions
Table 4: ROLL ablation study.
['[BOLD] Method', '[BOLD] Vis.', '[BOLD] Text.', '[BOLD] Temp.', '[BOLD] Know.', '[BOLD] All']
Ablation study We perform an ablation study to measure the contribution of each branch. When a single branch is used, the observe branch gets the worst overall accuracy and the recall branch performs the best. This is consistent with the types of questions in the dataset, with 22% being visual-based and 63% being knowledge-based. The read branch gets the best performance in the text-based questions (i.e., about the subtitles), and the recall branch gets the best accuracy in the knwoledge-based questions (i.e., about the storyline). When the observe branch is combined with other branches it consistently contributes to improve the results. Again, this result strongly suggests that the generated scene descriptions do contain meaningful information for the task. The full model combining the three branches performs the best. Fusion Methods Comparison We also study the performance of our proposed MW mechanism and compare it against several fusion methods. Given the three prediction scores from each of the branches, Average and Maximum compute the average and maximum score, respectively. The Self-att method implements a self-attention mechanism based on the Transformer outputs, and the QA-att mechanism attends each of the modality predictions based on the BERT representation of the question and candidate answers. The FC w/o MW predicts the answer scores by concatenating the scores of the three branches and feeding them into a linear layer, and FC w/ MW builds our proposed MW mechanism on top. The results show that most of the methods fail at properly fusing the information from the three branches, This is probably because the fusion of the different modalities incurs in information loss. Our MW mechanism, in contrast, successfully balances the contribution from the three branches. Fig. When the MW is not used (βω=1) the model obtains the worst performance. Likewise, when the loss contribution from the final prediction is too weak (βω<0.6), the model is not able to fuse the information correctly.