Are we asking the right questions in MovieQA?
Table 4: Experiments with our QA only model (for movies+subtitle task) with different amount of movie plots used for training Word2Vec (W2V). This table shows the importance of different word embeddings. Generic word embeddings, like Google’s (row 2) gives really poor accuracy. And using a better word embedding (row 5) can give really high accuracy, even without training the QA only model. When we use only val movie plots (row 3) we get good val accuracy but bad train accuracy and vice-versa. Highest accuracy is achieved when we use plots from train+val movies (row 5). Adding movies not in the dataset (row 9), results in degradation of accuracy. Even though same data are used for first and last row, the results differ because of slightly different hyper-parameters.
['#', 'W2V', 'Movie plots for training w2v', 'Train accuracy (w/o fine-tune)', 'Train accuracy', 'Val accuracy (w/o fine-tune)', 'Val accuracy']
Google and MovieQA word2vec: We plot performance both with and without fine-tuning our linear weighting layer. Google w2v performs poorly and close to chance (20%, second row), even after fine-tuning. This is likely because movie-specific words are missing in its vocabulary. MovieQA w2v itself gives about 38.71% accuracy (first row) after fine-tuning. Our word2vec: We now explore the effect of using subsets of movie plots to train w2v: ’train’, ’val’, and ’gen’ refer to plots from the train, val, and 1400-(train+val) movies respectively. This is notable because this system is not trained on any question-answer pairs. Finally, Wikiwords (training w2v on ’train+val’) leads to the best performance (49.88%). Hence, just using plots which are part of the dataset leads to the best accuracy and adding additional movie plots from the general population degrades performance. Our method naturally generates a partition of the data that is free of trivial language biases and can potentially be used for further progress in video-language modeling. We consider the QA’s which can be correctly answered by our WikiWord embedding model as the biased QA’s and the rest as the unbiased QA’s. In order to ensure that we don’t overfit to the data when finding these biased questions from the training set, we use the predictions of our untrained model. That is we take our QA only model which uses the pretrained word2vec, and we do not further train it with question-ground truth answer pairs. and so we drop all these QA ’s. Our hypothesis is that these questions are the really easy and the most biased ones, and video based models can solve without actually needing context from videos. To ensure that our unbiased dataset is competitive for multiple models, we show the performance of our QA-only model and that of the baseline QA-only model proposed in TVQA [lei2018tvqa] dataset. We observe the unbiased subset is harder for both models and results in close to chance level accuracy (20%). The unbiased subset hence provides a quick fix to the dataset.