Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions
Table 2: Evaluation on KnowIT VQA test set.
['[BOLD] Method', '[BOLD] Encoder', '[BOLD] Data  [BOLD] Dialog', '[BOLD] Data  [BOLD] Vision', '[BOLD] Data  [BOLD] Know.', '[BOLD] Accuracy  [BOLD] Vis.', '[BOLD] Accuracy  [BOLD] Text.', '[BOLD] Accuracy  [BOLD] Temp.', '[BOLD] Accuracy  [BOLD] Know.', '[BOLD] Accuracy  [BOLD] All']
Evaluation on KnowIT VQA We compare ROLL against the latest reported results on the KnowIT VQA dataset: TVQA and four different models in ROCK. ROCK reports results using four different visual representations: ResNet features, visual concepts, list of characters, and generated captions. Main findings are summarised as: Overall, ROLL outperforms previous methods in all the question categories by a large margin, with 6.3% improvement on the overall accuracy with respect to the best performing ROCK. When comparing the visual representations, our proposed video descriptions contain more semantic information than previous methods, improving visual-based questions by at least 6.4%. Specially, the boost in performance in visual-based questions with respect to standard captioning (ROCK Captions) or visual concepts (ROCK Concepts, TVQA) validates our unsupervised video descriptions as the best representation for this task. Additional evidence of the superior performance of our proposed unsupervised descriptions is shown when ROLL uses human annotations as external knowledge. Although the overall performance is lower because ROLL-human is not optimised to exploit this kind of information, on the visual-based questions our method improves best previous work by 5.4%. As the same source of knowledge is used, the superior performance can only be due to the contribution of our proposed visual representations. On the knowledge-based samples, our method based on plot summaries outperforms task-specific human annotations by 6.7%, even when less annotations are required. This implies that the proposed slicing mechanism in the recall branch successfully extracts the relevant information from the long documents provided as external knowledge. When compared against human performance, ROLL is 18% behind masters accuracy (humans that have watched the show) and it is closer to rookies non-knowledge accuracy (humans that have never watched the show). This shows how challenging this task is, with still plenty room for improvement.