Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension
Table 2: Percentage of questions in which the plot sentences containing the clues for the answer are ranked highest according to the model’s sentence attention distribution (relative to its selected answer) on the validation set (averaged results of nine models).
['Systems', 'CNN', 'RNN-LSTM']
The sentence attention allows us to get more insight into the models’ inner state. For example, it allows us to check whether the model actually focuses on relevant sentences in order to answer the questions. The MovieQA dataset provides human annotations of the minimal set of plot sentences required to answer a question. In average, 1.15/1.11 sentences in the training/validation set are marked as containing the clue to the answer. We leverage this information and compute the ranks of these relevant plot sentences according to the models’ sentence attention distribution. We extract the plot sentence relevance scores after the sentence-level comparison operation as average of Tsq and Tsaj, where aj corresponds to the selected answer of the model. Identifying the relevant sentences is an important success factor: Relevant sentences are ranked highest only in 35% of the incorrectly solved questions.