MovieQA: Understanding Stories in Movies through Question-Answering
Table 5: Accuracy for Text-based QA. Top: results for the Searching student with cosine similarity; Middle: Convnet SSCB; and Bottom: the modified Memory Network.
['Method', 'Plot', 'DVS', 'Subtitle', 'Script']
Cosine similarity with window. Using the plots to answer questions outperforms other sources (subtitles, scripts, and DVS) as the QAs were collected using plots and annotators often reproduce words from the plot. SkipThoughts perform much worse than both TF-IDF and Word2Vec which are closer. We suspect that while SkipThoughts are good at capturing the overall semantics of a sentence, proper nouns – names, places – are often hard to distinguish. Fig. TF-IDF and Word2Vec perform considerably well , however, we see a larger difference between the two for “Who” and “Why” questions. “Who” questions require distinguishing between names, and “Why” answers are typically long, and mean pooling destroys semantics. In fact Word2Vec performs best on “Where” questions that may use synonyms to indicate places. SkipThoughts perform best on “Why” questions where sentence semantics help improve answering. SSCB. Here, we present additional results combining all text representations (SSCB fusion) via our CNN. We split the train set into 90% train / 10% dev, such that all questions and answers of the same movie are in the same split, train our model on train and monitor performance on dev. Both val and test sets are held out. During training, we also create several model replicas and pick the ones with the best validation performance. Ignoring the case of plots, the accuracy is capped at about 30% for most modalities showing the difficulty of our dataset. The original MemN2N which trains the word embeddings along with the answering modules overfits heavily on our dataset leading to near random performance on val (∼20%). However, our modifications help in restraining the learning process. Using plot synopses, we see a performance closer to SSCB with Word2Vec features. However, in the case of longer stories, the attention mechanism in the network is able to sift through thousands of story sentences and performs well on DVS, subtitles and scripts. This shows that complex three-way scoring functions are needed to tackle such QA sources. In terms of story sources, the MemN2N performs best with scripts which contain the most information (descriptions, dialogs and speaker information).