Are we asking the right questions in MovieQA?
Table 6: Comparison of performance on different splits of MovieQA dataset for 2 different QA only models. The first row shows the original dataset. The second row shows the subset of original dataset which is biased i.e. our QA only model is able to correctly answer them. The third row is the subset which our QA only model is unable to correctly answer, resulting in chance level accuracy. These are the unbiased QAs and hence is the hardest split, and which would need information from videos and subtitles.
['Type', 'Our model', 'TVQA baseline [lei2018tvqa]']
Our method naturally generates a partition of the data that is free of trivial language biases and can potentially be used for further progress in video-language modeling. We consider the QA’s which can be correctly answered by our WikiWord embedding model as the biased QA’s and the rest as the unbiased QA’s. In order to ensure that we don’t overfit to the data when finding these biased questions from the training set, we use the predictions of our untrained model. That is we take our QA only model which uses the pretrained word2vec, and we do not further train it with question-ground truth answer pairs. and so we drop all these QA ’s. Our hypothesis is that these questions are the really easy and the most biased ones, and video based models can solve without actually needing context from videos. To ensure that our unbiased dataset is competitive for multiple models, we show the performance of our QA-only model and that of the baseline QA-only model proposed in TVQA [lei2018tvqa] dataset. We observe the unbiased subset is harder for both models and results in close to chance level accuracy (20%). The unbiased subset hence provides a quick fix to the dataset.